Add RTMPS relay support #344

mannyamorim · 2022-03-14T02:07:48Z

This is an attempt to add support for the relay module to directly connect to RTMPS targets. This is valuable for security and is required for certain streaming services (Facebook). Please test if you can, feedback is welcome 👍

Notes

Implemented a number of new SSL directives which are patterned after the ngx_http_proxy_module using the prefix rtmp_relay_. The directives follow the same syntax as the proxy module. Details are added to the doc folder.
Added two new parameters for relay targets to allow you to override certain behaviors. This is also documented.
A lot of code is patterned after the ngx_http_proxy_module and the ngx_http_upstream_module.
All of the new code is conditionally compiled using a new macro NGX_RTMP_SSL which is an alias for NGX_OPENSSL.
The only change to existing code is moving the ngx_rtmp_relay_app_conf_t to the header file.
Also included one bugfix discovered while testing, which is a segfault if the initial push fails. Details in the commit description for the fix.
Added support for the notify module to use RTMPS when receiving a 3xx response in an on_publish callback.
Added support for the notify module to resolve a hostname using DNS asynchronously when receiving a 3xx response in an on_publish callback.

Sample Configuration

I provide a sample configuration file below to demonstrate some of the possible options. The purpose of this config is a multi streaming server with one application that will push to YouTube and Facebook live. You will need to replace the xxxx with your streaming key.

load_module modules/ngx_rtmp_module.so;

worker_processes 1;

events {
    worker_connections 128;
}

http {
    server {
        listen 8080;

        location /facebook {
            rewrite ^.*$ rtmps://live-api-s.facebook.com/rtmp/****? permanent;
        }

        location /youtube {
            rewrite ^.*$ rtmps://a.rtmps.youtube.com/live2/****? permanent;
        }
    }
}

rtmp {
    resolver 1.1.1.1 8.8.8.8 ipv6=off;
    resolver_timeout 30s;

    rtmp_relay_ssl_protocols TLSv1.2 TLSv1.3;
    rtmp_relay_ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;
    rtmp_relay_ssl_server_name on;
    rtmp_relay_ssl_verify on;
    rtmp_relay_ssl_verify_depth 5;
    rtmp_relay_ssl_trusted_certificate /etc/ssl/certs/ca-certificates.crt;

    notify_relay_redirect on;
    drop_idle_publisher 10s;

    server {
        listen 1935;

        application test {
            live on;
            drop_idle_publisher 10s;

            push rtmps://a.rtmps.youtube.com/live2/xxxx;
            push rtmps://live-api-s.facebook.com/rtmp/xxxx;
        }

        application test2 {
            live on;

            push rtmp://127.0.0.1/facebook;
            push rtmp://127.0.0.1/youtube;
        }

        application facebook {
            live on;

            on_publish http://127.0.0.1:8080/facebook;
        }

        application youtube {
            live on;

            on_publish http://127.0.0.1:8080/youtube;
        }
    }
}

cc: #165 arut#1619 arut#747 arut#1605 arut#1408 arut#1587 arut#1604 arut#1397 arut#457 arut#232 arut#191 arut#68

When the initial push to a target fails in the relay module we can segfault since the relay ctx is NULL. THe segfault occurs in the if which sets up the timer for the push_reconnect method. In this patch we set the context using the existing create local ctx method passing in a NULL target as the target is unused in this method.

yildizsc · 2022-03-27T14:19:41Z

@mannyamorim This is great work. When this would be merged into the main repository?

mannyamorim · 2022-03-27T18:55:57Z

@mannyamorim This is great work. When this would be merged into the main repository?

Thanks @yildizsc. Merging is up to @sergey-dryabzhinsky.

When the notify module receives a 3xx response to the on_publish callback we will now support sending the stream to an rtmps target and we will support resolving the domain name.

RainNameless · 2022-04-09T20:30:12Z

Does it support tiktok?
rtmps://push-rtmp-l11-va01.tiktokcdn.com:443/game

mannyamorim · 2022-04-10T01:14:26Z

@RainNameless I've verified the code against Youtube and Facebook, but it should support any RTMPS target. Please test and let me know if there are any issues with Tiktok.

Be1erafon · 2022-04-13T06:47:05Z

@mannyamorim Tiktok and instagram works! Thanks for adding rtmps!

Be1erafon · 2022-04-14T07:34:49Z

Telegram doesn't work. Any ideas what is the reason?

server {
    listen 1935;
    application telegram {
        live on;
        push rtmps://dc4-1.rtmp.t.me/s/1234567890:xxxxxxxxxxxxxxxxx;  
    }
}

relay: SSL certificate does not match "dc4-1.rtmp.t.me/s/1234567890", client: dc4-1.rtmp.t.me/s/1234567890:xxxxxxxxxxxxxxxxx, server: ngx-relay

mannyamorim · 2022-04-14T12:48:48Z

@Be1erafon it looks like the SSL name is being calculated incorrectly. This is used for the TLS SNI field and for certificate hostname validation hence the failure to validate. This is happening since there is a colon in the URL that is not denoting a port number. Apparently this is legal, so I will need to tweak the code a bit to handle these cases correctly. I anticipate that I can have a fix available sometime tomorrow.

As a stopgap measure you could set: rtmp_relay_ssl_verify off; to disable any certificate validations.

Remove the custom logic for computing the SSL name as it is unnessary and replace it with the standard URL parsing logic found in NGINX. Also add one missing #undef directive.

mannyamorim · 2022-04-15T02:06:07Z

@Be1erafon I've pushed a commit which should fix the issue you were experiencing. Please test again and let me know the results.

Be1erafon · 2022-04-15T07:46:14Z

@mannyamorim Connection is working.
The video stream in the telegram does not go. According to the statistics, there is a connection, the data is being sent.

Doesn't work

application telegram {
    live on;
    push rtmps://dc4-1.rtmp.t.me/s/1234567890:xxxxxxxxxxxxxxxxx; 
}

Works via ffmpeg

application telegram-ffmpeg {
    live on;
    exec_push ffmpeg -re 
        -i rtmp://127.0.0.1/telegram-ffmpeg 
        -c:v copy -c:a copy 
        -f flv rtmps://dc4-1.rtmp.t.me/s/1234567890:xxxxxxxxxxxxxxxxx;
}

Looks like Telegram has changed something. Worked with nginx a few days ago.
ps: youtube, tiktok, instagram by rtmps works.

mannyamorim · 2022-04-15T23:52:04Z

@Be1erafon thanks for letting me know. I'll try to work out what the difference is between the NGINX module and the FFMPEG implementation. Will post back once I have a better understanding of this problem.

Rosstarz · 2022-05-07T14:37:32Z

this is really cool.

could you give me a quick guide on how to install nginx with your version of nginx-rtmp module? I tried downloading your repo ssl branch

git clone -b ssl https://github.com/mannyamorim/nginx-rtmp-module.git

and then following nginx guide https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-open-source/#sources and then do

./configure --add-module=/path/to/nginx-rtmp-module make make install

but when I run nginx with your configuration I get

nginx: [emerg] dlopen() "/usr/share/nginx/modules/ngx_rtmp_module.so" failed (/usr/share/nginx/modules/ngx_rtmp_module.so: cannot open shared object file: No such file or directory) in /etc/nginx/nginx.conf:6

any ideas?

sergey-dryabzhinsky · 2022-05-07T15:53:33Z

You should build dynamic module:

./configure --add-dynamic-module=/path/to/nginx-rtmp-module
make make install

mannyamorim · 2022-05-08T02:56:03Z

@Rosstarz you can read a bit about dynamic modules in NGINX here: https://www.nginx.com/resources/wiki/extending/converting/.

You can use the commands that @sergey-dryabzhinsky posted to build as a dynamic module. If you would like to use it as a static module instead (the way you were originally trying), you will just need to remove the line at the top of the sample config file load_module modules/ngx_rtmp_module.so;. When compiled as a static module the directives will be available without the need to load any additional code.

yildizsc · 2022-05-21T20:34:23Z

Hi @mannyamorim,

Thanks again for this great afford. This setup is actually working for AWS IVS. But I need to make rtmp_relay_ssl_verify off to make it work while using self-signed SSL.

Here is my configuration:

  rtmp_relay_ssl_protocols TLSv1.2 TLSv1.3;
  rtmp_relay_ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;
  rtmp_relay_ssl_server_name on;
  rtmp_relay_ssl_verify off;
  rtmp_relay_ssl_verify_depth 5;
  rtmp_relay_ssl_trusted_certificate /certs/domain.crt;

  notify_relay_redirect on;

And here is the code which proceses self-signed SSL:

mkdir /certs && cd /certs && openssl req \
  -newkey rsa:2048 -nodes -keyout domain.key -subj "/C=FR/O=krkr/OU=Domain Control Validated/CN=*.domain.com" \
  -x509 -days 365 -out domain.crt

Do I need an authoritative SSL?

mannyamorim · 2022-05-22T02:32:28Z

@yildizsc from Nginx's point of view there is no difference between authoritative SSL and self-signed certificates. Both should work just fine.

In this case, I believe you need to create a CA certificate (self-signed of course) first and then use it to sign a client certificate for your application. The CA certificate would then be what you pass into the rtmp_relay_ssl_trusted_certificate directive.

If you're still having trouble, please post the error logs that you get. If nothing descriptive, please add the debug option to the error_log directive. I'll need to see the specific error message to understand why the SSL verification is not working.

yildizsc · 2022-05-22T06:32:09Z

@mannyamorim

2022/05/22 06:24:23 [error] 14#14: *10 relay: SSL certificate verify error: (20:unable to get local issuer certificate), client: somertmp/live/token, server: ngx-relay
2022/05/22 06:24:23 [error] 14#14: *10 relay: SSL certificate verify error: (20:unable to get local issuer certificate), client: somertmp/live/token, server: ngx-relay

This is what I get with debug mode.

Btw, could you elaborate on creating the certificate first? Does it matter? I could get the point.

mannyamorim · 2022-05-23T14:43:03Z

@yildizsc please ignore my earlier comments about needing to have a CA certificate. It appears that this is not necessary.

The error message "unable to get local issuer certificate" didn't turn up anything terribly useful. It just seems to be a generic message that the openssl library can't load your certificate. I just ran a quick test with a single self signed certificate and it seems to be working fine. If you try out this setup, hopefully you can work towards your setup and see where the difference is that's causing the issue. Also, could you post your full Nginx config file (with anything sensitive removed/replace with dummy values), hopefully that can help to shed some more light on the problem.

Test setup was Nginx relaying to an Stunnel instance with a self signed certificate with certificate generated with the following command:

openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -sha256 -days 365 -nodes -subj '/CN=localhost'

Stunnel conf:

[RTMP IN Server]
accept = 127.0.0.1:1937
connect = a.rtmp.youtube.com:1935
cert = /etc/stunnel/cert.pem
key = /etc/stunnel/key.pem

Nginx conf:

rtmp {
	rtmp_relay_ssl_protocols TLSv1.2 TLSv1.3;
	rtmp_relay_ssl_ciphers ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;
	rtmp_relay_ssl_server_name on;
	rtmp_relay_ssl_verify on;
	rtmp_relay_ssl_verify_depth 5;
	rtmp_relay_ssl_trusted_certificate /etc/stunnel/cert.pem;

	server {
		access_log /nginx/access.log;
		listen 1935;

		application test3 {
			live on;
			push rtmps://localhost:1937/live2/xxxx;
		}
	}
}

Please note that the CN (Common Name) of the certificate needs to match the name used in the push command in Nginx. Alternatively the SAN field could be used as well. In this case I've set the CN to localhost but you can use any domain.

kuznetcoff777 · 2022-06-07T09:47:29Z

@Be1erafon thanks for letting me know. I'll try to work out what the difference is between the NGINX module and the FFMPEG implementation. Will post back once I have a better understanding of this problem.

Hi @mannyamorim, if any success with this? Faced the same issue with telegram...
When use push rtmps - no erros in debug, just see as stream goes there (tcpdump also prove it), but telegram says he does not see it.

mannyamorim · 2022-06-13T23:37:18Z

@Be1erafon thanks for letting me know. I'll try to work out what the difference is between the NGINX module and the FFMPEG implementation. Will post back once I have a better understanding of this problem.

Hi @mannyamorim, if any success with this? Faced the same issue with telegram... When use push rtmps - no erros in debug, just see as stream goes there (tcpdump also prove it), but telegram says he does not see it.

Hi @kuznetcoff777, sorry for the delay. I haven't been able to figure out this mystery yet. I ran a test with the NGINX stream module to see if it is an issue with the SSL code in this PR. I used the following configuration:

worker_processes  1;

error_log  xxxxx;

events {
	worker_connections  1024;
}

stream {
	server {
		listen 5000;

		proxy_pass dc1-1.rtmp.t.me:443;
		proxy_ssl on;
		proxy_ssl_protocols TLSv1.2 TLSv1.3;
		proxy_ssl_server_name on;
		proxy_ssl_verify on;
		proxy_ssl_verify_depth 5;
		proxy_ssl_trusted_certificate /etc/ssl/certs/ca-certificates.crt;
	}
}

rtmp {
	server {
        	access_log xxxxx;

		listen 1935;

		application test {
			live on;
			push rtmp://127.0.0.1:5000/s/xxxxx;
		}
	}
}

Essentially there is a single push command that pushes to a local endpoint which is then proxied to the SSL destination at Telegram using the NGINX SSL stream module. This failed with the same symptoms as with this PR. The debug error log shows that the RTMP session is operating normally but nothing is displayed in Telegram. This looks to be an issue in the main RTMP module and not with this PR. The guts of the RTMP protocol is really outside of my knowledge, so I don't think that I will be able to debug further.

I have logged this issue as #350. At this point I don't consider it an issue with this PR and likely won't be looking into it any further. Please let me know if you have a different opinion or further information.

bertin0 · 2022-10-12T12:15:49Z

I see that you have added a dockerfile in your fork. Could you provide a basic tutorial of how to run it?

mannyamorim · 2022-10-17T13:07:31Z

I see that you have added a dockerfile in your fork. Could you provide a basic tutorial of how to run it?

@bertin0 the Dockerfile that I have added to my fork is on a different branch, and I am not planning on submitting a PR for it, so not really relevant here. That being said, the usage is very simple. Basically, the container is just a standard nginx docker container plus the rtmp module added as a dynamic module. You can refer to the documentation for the nginx docker container here: https://hub.docker.com/_/nginx. I would recommend just mounting the config file directly. For the configuration file, you can refer to the documentation in the PR.

haerong22 · 2023-04-27T13:26:47Z

@mannyamorim Can you pass the stream key dynamically?

mannyamorim · 2023-04-27T13:37:39Z

@mannyamorim Can you pass the stream key dynamically?

@haerong22 I am assuming that you want to relay an RTMP stream to an RTMPS target with a dynamic stream key. This is possible using the notify module and the on_publish callback. You can refer to the existing documentation. In this PR I added support for you to use a URL that contains a domain name instead of just a static IP.

Essentially what happens here is that you can configure the module to execute a callback to a certain URL. If that URL returns a redirect then the returned URL with the stream key appended to it, is used as the target for the rtmp relay.

If you take a look at the sample configuration for the PR. The application test2 shows how you can build a relay system with multiple targets where all URLs and keys are dynamic. In this case the actual URLs with the keys are served by Nginx in the http block, however you can use anything you want to serve the URLs with the keys dynamically.

haerong22 · 2023-04-27T13:56:21Z

@mannyamorim omg! so fast... thanks. but i can't understand ...

i want OBS -> NGINX -> YOUTUBE
how can i pass youtube stream key???
i try OBS -> NGINX : rtmp://localhost:1935/test/key?youtube=youtube-stream-key

i want like this

location /youtube {
            rewrite ^.*$ rtmps://a.rtmps.youtube.com/live2/$arg_youtube? permanent;
 }

rtmp {
    ...
    server {
         
         application test {
                ...
                push rtmp://127.0.0.1:1935/youtube;
                ...
         }

         application youtube {
            live on;

            on_publish http://127.0.0.1:8080/youtube;
        }
    }
}

mannyamorim · 2023-05-08T00:54:28Z

@haerong22 Sorry, that configuration is not directly supported by this module, however with a bit more work you could build something like this.

When you use the on_publish command, it appears that any arguments that you use should get passed through. Maybe if you try something like https://github.com/calio/form-input-nginx-module you can get the rewrite command to read in the arguments into a variable so you can do something like your example above.

This is just an idea, not something tested or supported by me, so you will need to experiment a bit.

spjoes · 2023-05-30T23:43:00Z

May need a different pull request but while you're at it with port 443 and rtmps, why not allow https on notify (like on_publish, on_play, on_publish_done, etc.)?
I've tried adding it myself before but keep getting "invalid port in url" and such. Not sure why.
Even tried one of the other pull requests on nginx-rtmp-module. Same issue.

mannyamorim · 2023-05-31T00:43:07Z

May need a different pull request but while you're at it with port 443 and rtmps, why not allow https on notify (like on_publish, on_play, on_publish_done, etc.)? I've tried adding it myself before but keep getting "invalid port in url" and such. Not sure why. Even tried one of the other pull requests on nginx-rtmp-module. Same issue.

@spjoes it is not trivial to add https support, rather it would require a fair amount of new code and testing. At this point I have no plans to add any such support to my fork. I understand that it would be more convenient, however you can simply have Nginx function as an RP for any https endpoints that you need to call out to, so this limitation shouldn't really impact the functionality of the module.

spjoes · 2023-05-31T00:47:27Z

@spjoes it is not trivial to add https support, rather it would require a fair amount of new code and testing. At this point I have no plans to add any such support to my fork. I understand that it would be more convenient, however you can simply have Nginx function as an RP for any https endpoints that you need to call out to, so this limitation shouldn't really impact the functionality of the module.

I was actually never able to get a RP working. Tried for a few hours and figured it'd be a better, cleaner, and (at the time) a quicker fix to just have https as a feature. I'll continue to try with proxy_pass and such but just native https support would be a great feature

mannyamorim added 3 commits March 13, 2022 11:47

Initial support for SSL in the rtmp relay module

88b7951

Add documentation for new SSL directives in the relay module

366218c

Add DNS & RTMPS support for the notify module on_publish

fa7bdcf

When the notify module receives a 3xx response to the on_publish callback we will now support sending the stream to an rtmps target and we will support resolving the domain name.

fix SSL name computed incorrectly in certain cases

14a38fb

Remove the custom logic for computing the SSL name as it is unnessary and replace it with the standard URL parsing logic found in NGINX. Also add one missing #undef directive.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add RTMPS relay support #344

Add RTMPS relay support #344

mannyamorim commented Mar 14, 2022 •

edited

yildizsc commented Mar 27, 2022

mannyamorim commented Mar 27, 2022

RainNameless commented Apr 9, 2022

mannyamorim commented Apr 10, 2022

Be1erafon commented Apr 13, 2022

Be1erafon commented Apr 14, 2022

mannyamorim commented Apr 14, 2022 •

edited

mannyamorim commented Apr 15, 2022

Be1erafon commented Apr 15, 2022

mannyamorim commented Apr 15, 2022

Rosstarz commented May 7, 2022 •

edited

sergey-dryabzhinsky commented May 7, 2022

mannyamorim commented May 8, 2022

yildizsc commented May 21, 2022

mannyamorim commented May 22, 2022

yildizsc commented May 22, 2022

mannyamorim commented May 23, 2022 •

edited

kuznetcoff777 commented Jun 7, 2022

mannyamorim commented Jun 13, 2022

bertin0 commented Oct 12, 2022

mannyamorim commented Oct 17, 2022

haerong22 commented Apr 27, 2023 •

edited

mannyamorim commented Apr 27, 2023

haerong22 commented Apr 27, 2023 •

edited

mannyamorim commented May 8, 2023

spjoes commented May 30, 2023

mannyamorim commented May 31, 2023

spjoes commented May 31, 2023 •

edited

Add RTMPS relay support #344

Are you sure you want to change the base?

Add RTMPS relay support #344

Conversation

mannyamorim commented Mar 14, 2022 • edited

Notes

Sample Configuration

yildizsc commented Mar 27, 2022

mannyamorim commented Mar 27, 2022

RainNameless commented Apr 9, 2022

mannyamorim commented Apr 10, 2022

Be1erafon commented Apr 13, 2022

Be1erafon commented Apr 14, 2022

mannyamorim commented Apr 14, 2022 • edited

mannyamorim commented Apr 15, 2022

Be1erafon commented Apr 15, 2022

mannyamorim commented Apr 15, 2022

Rosstarz commented May 7, 2022 • edited

sergey-dryabzhinsky commented May 7, 2022

mannyamorim commented May 8, 2022

yildizsc commented May 21, 2022

mannyamorim commented May 22, 2022

yildizsc commented May 22, 2022

mannyamorim commented May 23, 2022 • edited

kuznetcoff777 commented Jun 7, 2022

mannyamorim commented Jun 13, 2022

bertin0 commented Oct 12, 2022

mannyamorim commented Oct 17, 2022

haerong22 commented Apr 27, 2023 • edited

mannyamorim commented Apr 27, 2023

haerong22 commented Apr 27, 2023 • edited

mannyamorim commented May 8, 2023

spjoes commented May 30, 2023

mannyamorim commented May 31, 2023

spjoes commented May 31, 2023 • edited

mannyamorim commented Mar 14, 2022 •

edited

mannyamorim commented Apr 14, 2022 •

edited

Rosstarz commented May 7, 2022 •

edited

mannyamorim commented May 23, 2022 •

edited

haerong22 commented Apr 27, 2023 •

edited

haerong22 commented Apr 27, 2023 •

edited

spjoes commented May 31, 2023 •

edited