Add SAT>IP support (remote network tuners)

[perexg]

This code adds initial support for remote SAT>IP tuners (servers). I touched slightly other parts, too.

[perexg]

A comment for users with GSS.BOX and compatible (Inverto Airscreen Server, Telestar Digibit R1, Elgato EyeTV Netstream 4Sat and probably others):

Please, upgrade to firmware V1.16.0.120 available at http://inverto.tv/support - Firmware tab - IDL 400s product .

The previous firmwares have so many limitations (and probably won't work correctly).

[adamsutton]

@perexg Not looked at this in detail yet, but looks really interesting. I'd ignored SAT>IP for a long time as I couldn't really see a big benefit. But that GSS box looks really cool for the price / tuners. I'm considering getting one myself to setup a test/dev system.

Couple of minor points though:

1. Says I can't auto-merge from here, which usually implies there could be conflicts on merge, maybe you could rebase this on top of current master?
2. Is there any chance some of the RTSP code used in here could be split out into a generic RTSP client module? It's just I've also been keen to integrate @Glandos RTSP/IPTV PR, but just never found the time (as it's no longer compatible post DVB rewrite). And it would seem sensible to try and common this stuff up where possible.

Adam

[clandmeter]

@perexg What is your current experience with the GSS.BOX? I'm really interested to buy the GSS.BOX like you have mentioned. The only thing currently holding me back is the confirmation if it will support decryption inside of TVH. Did you already run some test to see if CWC/CAPMT will work?

[perexg]

I rebased this branch. @Glandos code uses curl. I wrote own simple implementation, because the SAT>IP protocol is very simple, but it may be extended... Also, I found using the curl library a bit overkill for tvh purposes and it depends on the NSPR library. Do you have any objections against the idea to write own http/rtsp client code?

[perexg]

@clandmeter : With the V1.16.0.120 firmware seems usable and stable. The decryption is not a issue - I'm testing it with a Cryptoworks card - it's totaly another layer above the SAT>IP specification.

From my first testings - the tuners seem good - better than DVBSKY S952 dual tuner PCI-E cards I am using currently in my home server and tuning is fast - about 100-150ms.

[EricV]

@perexg I do not understand you sentence "The decryption is not a issue". Reading from data sheet it seems to stream over LAN only unencrypted channels. I doubt it will decrypt the content and then send it unencrypted over LAN meaning that you should have a decoder with a card on the receiver side that should be handled by tvheadend. This is typically the problem you have with IPTV and not all cards or crypto algorithm are supported by oscam. This is also true for encrypted TNT tv...

[perexg]

@EricV : It's not really an issue - SAT>IP is not about it. The box behaves like a remote tuner and sends all contents from the satellite or terresterial multiplex to the ethernet network including the information howto decrypt the contents when user has the proper smartcard from the contents provider. And yes, the decryption must be processed on the client side (in this case the tvheadend server with properly configured CSA connections to a server reading decryption keys from smart cards - oscam, newcamd ...). In other words, GSS.BOX and such devices are really remote ethernet tuner like you put an PCI or USB card to your Linux box without any CAM module.

But on other side, the vendors may implement the TS stream decryption in the SAT>IP servers (so a slot for a smart card / CI module may be there - I think that Octopus NET supports CI modules - http://shop.digital-devices.de ).

[perexg]

Just to clarify - GSS.BOX supports only DVB-S/S2, but other boxes can support DVB-T/T2 - the SAT>IP specification covers also terresterial DVB.

[EricV]

@perexg just to make it clear, I do not have anything against introducing SAT>IP support. I do have hdhomerun devices for DVB-T (but they have no DVB-S) and I'm quite pleased with them even if protocol is non standard.

[adamsutton]

@perexg I've no issue with replacing CURL usage with our own HTTP client, I did sort of start this, but I really couldn't be bothered with the SSL stuff given there was an existing library that most people (given we only officially support ubuntu/debian) will have easy access to.

[perexg]

@adamsutton : Do you think that somebody will stream through SSL ?

[adamsutton]

@perexg what, apart from me :p

In reality, it's not a massive deal, but we do have a requirement for HTTPS (client) elsewhere. So if we were thinking of removing CURL, then we would need to support HTTPS. And if we can't add that support and thus require CURL for that, it somewhat negates the original intent.

If that wasn't completely confusing!

[clandmeter]

@perexg you mention the "Telestar Digibit R1" and GSS.BOX. Do they have exactly the same hardware? On amazon.de the Telestar is about 10 Euro cheaper, wondering which i should buy.

[Jalle19]

Why not reuse existing code and use curl?

[perexg]

@clandmeter : I believe yes, the hardware is identical. It's developed by inverto.tv and rebranded . I checked the screenshots and firmware files and seems same as for my GSS.BOX .

[perexg]

@Jalle19 : The problem with curl is that it's only "callback" based and eventually it may stall (block) in some situations. I use different socket access in my simple RTSP client for SAT>IP without callbacks and shared poll() multiplexing in one thread. This thread handles RTSP, RTP, RTCP and piped commands from other threads using standard poll() multiplexing. The avoidance of the blocking is critical.

[adamsutton]

I agreed with @perexg on this point, while curl can be made to work, it definitely took me a while to get things working in a sensible fashion (and even then its messy). I would have much preferred a much simpler API, though I think this is probably a bit harder to achieve (though not impossible) when you throw SSL into the mix and so must do processing on the incoming data before passing off to caller.

[perexg]

@adamsutton : It seems that the SSL inline processor may be implemented quite easily: http://funcptr.net/2012/04/08/openssl-as-a-filter-%28or-non-blocking-openssl%29/

[Jalle19]

I concur with @adamsutton that the RTSP code should be separated so it can be used for generic IPTV inputs (if that's not the case already, kinda hard to tell at a glance when there's only a single commit here). One interesting use-case would be to run an ArgusTV server on Windows and import those streams in tvheadend (Argus uses rtsp://). This way people with Windows-only tuners could still use tvheadend if they want to (they could run it in a VM if only IPTV inputs are necessary). Obviously there are other valid use-cases too, this is just something I came up with just now.

[perexg]

I think that the RTSP common code should be separated, too. Actually satip_rtsp.c contains cca 30% possible common code and 70% SAT-IP specific code, because the RTSP implementation used in SAT-IP is really basic and we don't need to parse any formats (use the DESCRIBE command). Only the session and the connection info (UDP unicast/multicast) is maintained.

But RTSP is based on HTTP - this was the reason I asked Adam, if the http client engine can be written from scratch. I will probably come with a better solution if I can integrate the SSL support in a very easy way.

[perexg]

A code split to separate the SAT>IP code and fixes in other parts.

[Silverchairmfcu]

I got an issue during the compile of src/http/http_client.c, the cast seams to be incorrect in line 116.

src/http/http_client.c: In function âhttp_curl_socketâ:
src/http/http_client.c:116:19: error: cast from pointer to integer of different size [-Werror=pointer-to-int-cast]
src/http/http_client.c: In function âhttp_threadâ:
src/http/http_client.c:157:11: error: cast from pointer to integer of different size [-Werror=pointer-to-int-cast]
src/http/http_client.c:177:13: error: cast from pointer to integer of different size [-Werror=pointer-to-int-cast]
src/http/http_client.c: In function âhttp_client_initâ:
src/http/http_client.c:264:17: error: cast from pointer to integer of different size [-Werror=pointer-to-int-cast]
cc1: all warnings being treated as errors
make: *** [/home/pi/tv_satip/tvheadend-satip/build.linux/src/http/http_client.o] Fehler 1

[perexg]

@Silverchairmfcu the commit 5a72452 should fix this issue.

[perexg]

OK. I wrote the HTTP client code including support for SSL, chunked transfers and RTSP. Here is the updated satip branch.

[clandmeter]

@perexg I just received my GSS.BOX so I tried to compile your satip branch but i'm getting errors. I would like to note, I'm using http://www.musl-libc.org as libc.

edge-musl-x86_64:~/tvheadend-ip$ make
CC              src/version.o
CC              src/uuid.o
CC              src/main.o
CC              src/tvhlog.o
CC              src/idnode.o
CC              src/prop.o
CC              src/utils.o
CC              src/wrappers.o
CC              src/access.o
CC              src/dtable.o
CC              src/tcp.o
CC              src/udp.o
CC              src/url.o
src/url.c: In function 'urlparse':
src/url.c:109:87: error: incompatible types when assigning to type 'char[256]' from type 'char *'
     uri_copy(buf, path->text);
                                                                                       ^
src/url.c: At top level:
src/url.c:130:1: error: no previous prototype for 'urlparse_free' [-Werror=missing-prototypes]
 urlparse_free( void )
 ^
cc1: all warnings being treated as errors
Makefile:317: recipe for target '/home/clandmeter/tvheadend-ip/build.linux/src/url.o' failed
make: *** [/home/clandmeter/tvheadend-ip/build.linux/src/url.o] Error 1

[perexg]

I don't use URIPARSER library, let me know, if above commit works for you.

[clandmeter]

@perexg works! but bumped into next issue

CC              src/input/mpegts/linuxdvb/linuxdvb_en50494.o
CC              src/input/mpegts/satip/satip.o
CC              src/input/mpegts/satip/satip_frontend.o
src/input/mpegts/satip/satip_frontend.c: In function 'satip_frontend_input_thread':
src/input/mpegts/satip/satip_frontend.c:665:18: error: array type has incomplete element type
   struct mmsghdr msg[PKTS];
                  ^
src/input/mpegts/satip/satip_frontend.c:806:5: warning: implicit declaration of function 'recvmmsg' [-Wimplicit-function-declaration]
     tc = recvmmsg(lfe->sf_rtp->fd, msg, PKTS, MSG_DONTWAIT, NULL);
     ^
src/input/mpegts/satip/satip_frontend.c:665:18: warning: unused variable 'msg' [-Wunused-variable]
   struct mmsghdr msg[PKTS];
                  ^
Makefile:318: recipe for target '/home/clandmeter/tvheadend-ip/build.linux/src/input/mpegts/satip/satip_frontend.o' failed
make: *** [/home/clandmeter/tvheadend-ip/build.linux/src/input/mpegts/satip/satip_frontend.o] Error 1

[perexg]

Your libc library does not support the recvmmsg() syscall - http://man7.org/linux/man-pages/man2/recvmmsg.2.html . I will try to add an emulation code.

[perexg]

Please, try this update.

[clandmeter]

@perexg

CC              src/input/mpegts/satip/satip.o
CC              src/input/mpegts/satip/satip_frontend.o
src/input/mpegts/satip/satip_frontend.c: In function 'satip_frontend_input_thread':
src/input/mpegts/satip/satip_frontend.c:685:18: error: array type has incomplete element type
   struct mmsghdr msg[PKTS];
                  ^
src/input/mpegts/satip/satip_frontend.c:826:5: error: implicit declaration of function 'recvmmsg' [-Werror=implicit-function-declaration]
     tc = recvmmsg(lfe->sf_rtp->fd, msg, PKTS, MSG_DONTWAIT, NULL);
     ^
src/input/mpegts/satip/satip_frontend.c:685:18: error: unused variable 'msg' [-Werror=unused-variable]
   struct mmsghdr msg[PKTS];
                  ^
cc1: all warnings being treated as errors
Makefile:317: recipe for target '/home/clandmeter/tvheadend-ip2/build.linux/src/input/mpegts/satip/satip_frontend.o' failed
make: *** [/home/clandmeter/tvheadend-ip2/build.linux/src/input/mpegts/satip/satip_frontend.o] Error 1
make: *** Waiting for unfinished jobs....

[christophdb]

Hi hennoa and all others:
thanks for your support. It seems that wrong configuration values prevented my tvheadend from finding any sources. Now I got already 1.800 services and the system is scanning for the channel names. I think now I can handle it.
Best regards
Christoph

[ckarrie]

@perexg Hi perexg, is there a plan to add SAT>IP Server capability to Tvheadend? This would be useful for SAT>IP STBs without descrambling capabilities.

[perexg]

@ckarrie : It's on my todo list.

[ckarrie]

@perexg wonderful! no hurry, definitely a nice-to-have feature!

[clandmeter]

@perexg Im having lots of issues currently with SATIP. It seems somebody else is having the same errors as me (we both have gsbox). I cant find you on IRC, so I thought i would like a msg here. I have trace logs and core dumps for you if you can take a look.

[perexg]

@clandmeter : I'm actually 99.9% AFK until end of next week . I'll join to IRC after that.

[m4tt075]

Dear All,

What you are doing here is absolutely amazing! Thanks for all the efforts.
Does anyone of you actually know whether this SAT>IP implementation would also work with a multifeed system (2 quad LNBs or a twin multifeed LNB)? And if so, would that restrict the choice of the SAT>IP server that could be used (I would of course like to stick to those that you have implemented already if possible)?
Any advice appreciated. With best regards,
M4tt075

[MyFriendlyUserName]

@perexg I am trying to setup TVheadend with Elgato Netstream 4SAT, which is another SAT-IP server. Unfortunately I get the same error as @clandmeter in April:

2014-08-23 20:16:49.852 [ ERROR] satip: Cannot get http://192.168.200.224:81/description.xml: Falscher Medien-Typ
= german for "Wrong medium type"

If I open the upper URL in a browser I get an XML document.

Do you have an idea how to solve this?

[renne]

@m4tt075
As long as the SAT>IP-server follows the specification EN 50585:2014 you can use any SAT>IP-server. SAT>IP just outsources the tuners into an extra box which the client connects to via the standardized SAT>IP protocol. So, no hassle with hardware drivers anymore but the SAT>IP-server can send any Diseqc-command like before. On input-side SAT>IP-servers are just like set-top boxes with 2, 4 or 16 tuners. Most of them also support Unicable (EN 50494) and Quattro LNBs. Clients (like TVs, set-top boxes, TVHeadend, ...) can also handle multiple SAT>IP-servers only limited by the bandwidth of your network. So you can fiddle together any number and types of LNBs, MultiSwitches and SAT>IP servers. If you want to get rid of coax-cables completely, you can put one or more SAT>IP-LNBs onto your satelite dish. Currently there's only the Inverto iLNB which will be shipped on September, 19th by Reichelt.

[ullix]

I can highly recommend the Inverto 8 channel SAT>IP LNB with PoE Adapter, which is now broadly available from 330€ and up (http://www.heise.de/preisvergleich/inverto-idli-8che20-oopoe-osp-a1170426.html).

Works marvelously at least when combined with tvheadend. My first impressions reported here: https://tvheadend.org/boards/5/topics/13486

I have it running for a few weeks now. Some initial struggle with understanding the whole thing, but since then it just works!

[nurtext]

Telestar Digibit R1 works very well for me since a 3 months. Just in case you don't want or need a native SAT>IP LNB with PoE power. Costs just 125,00 €.

[perexg]

Note for GSS.BOX and compatible: It seems that engineers don't handled well the temperature constraints. Under some loads and some conditions (temperature, bad cooling), the gss.box can be unstable (small or massive drop-outs).

I replaced/added the heatsinks for all chips and added a little fan for the air circulation. The server seems pretty stable now even under a big load.

[ullix]

I am still using the "8 channel SAT>IP LNB with PoE Adapter" from Inverto, but my initial enthusiasm has taken some big hits recently. This thing in combination with tvheadend hangs so often that I sometimes can't even watch/record one single movie!

I have played with setting various settings on and off that are available under tvheadends TV-Adapters but to no avail. The symptoms are: tvheadend hangs, and must be brought back via stopping + starting, and also the tuner hangs, and the only thing working to bring it back is a power disconnect (done by pulling the ethernet cable from the POE adapter). Interestingly, while the tuner hangs, and makes even a "wget " time-out, the tuner can be pinged and responds properly and fast.

Of course, it could also mean that the tuner is somehow overloaded, has a bug, or a temperature problem, or whatever. However, I am now running the tuner with eight instances of vlc, blocking all 8 tuners (vlc does not seem to be able to use more than one mux per tuned frequency), 1 channel is Full-HD, 4 are HD, and 3 are SD. Total bitrate is slightly above 80Mbit/s. Each of the 8CPUs of an i7-4771 CPU @ 3.50GHz is used at ~50% load. That 's s as much load as I can generate now and it is running smoothly for now well over 30h.

I think it looks more like a tvheadend problem. Unfortunately, I don't know how to investigate further. I have a python script to ping and wget the tuner every minute, but of course it does not tell me, what if anything, tvheadend has done.

Is there a debug option to get a record of the things that tvheadend sends to the tuner? And is it possible to extend this such that tvheadend tests (by e.g. a wget to the tuner) that the tuner is still responding ? Of course, since tvheadend also freezes with whatever it does, a timeout may have to be built in too.

I'd be happy to do more testing, but I am out of options for now.

[ckarrie]

Hi @ullix: did you factory reset your Sat>IP LNB? This resolved 99% of all issues you described at my side. I didn't restarted the SAT>IP LNB since my last comment here (2 months ago). However I have some problems getting all HD Channels working (esp. crypted channels) crypted SD channels always works fine, then i need to restart Tvheadend.

[ullix]

@ckarrie : yes, restart and also factory reset done more than once.
And I tried it with "Activate smart tune mode (recommended) " in the tuner settings both on and off.
My Firmware Version is still : V1.20.0.71, as it was from the very beginning. Anyone having different firmware?

[perexg]

Even a bug in tvheadend cannot cause to hang the firmware in the box... You can do '--trace satip,httpc' to trace the control communication between tvheadend and the SAT>IP server or you may use any other network analyzer (tcpdump, wireshark) to grab this communication and see where communication is lost.

I know about the hang issue in tvh - I need to move the SAT>IP code to another thread outside the main lock, but if the SAT>IP server responds properly, this situation cannot occur. It's like you detach the live network cable.

[ullix]

The fact is that the tuner box does hang very firmly, and so far this was observed only when tvheadend was active! Still, even under this hang, you could ping the tuner. But not connect to it with wget or firefox or else.

I have activated satip and httpc via the web interace. We'll see.

[perexg]

@ullix : This box runs linux inside. The ping is absolutely different task in the linux kernel than other things. Inverto uses own user space application which contains HTTP server and SAT>IP server which seems to hang under some circumstances. And yes, tvheadend is very complex tool which might trigger totally different issues in the firmware. I think that there is some timing problem when new mux is tuned inside a time limit (fast mux change) - then their firmware (user space app) just hangs or so.. I would ask Inverto, but don't expect any success. My questions about the temperature stability for 400s resulted to reply that their engineers are not aware of this and they haven't seen it. It's all..

EDIT: Note that there is a timeout value in which ANY SAT>IP server must recover in the SAT>IP specification. It's usually 60 seconds.

[ckarrie]

@ullix Same Firmware Version. Leave "Activate smart tune mode (recommended)" turned on (don't know the magic behind that - but it works a lot better). To get a working SAT>IP LNB try to set following settings for each Tuner: "Next tune delay in ms (0-2000):" to 2000, check "Force teardown delay: ", uncheck "Send full PLAY cmd". For the IPLNB check "Full Mux Rx mode supported", check "addpids/delpids supported:", uncheck "PIDs in setup:", check "Force pilot for DVB-S2:". This works for me.

@perexg Time to build our own SAT>IP Servers with TVH running ;-)

[ullix]

Well, there we go again with yet another hang.

This is what the tvh log gave (with satip and httpc trace enabled):
2014-12-11 22:59:31.503 [ INFO]:mpegts: 12480V in SATIP - scan timed out
2014-12-11 22:59:31.503 [ INFO]:subscription: 1EB4: "scan" unsubscribing
2014-12-11 22:59:31.503 [ DEBUG]:satip: SAT>IP DVB-S Tuner #8 (10.0.0.71) - stopping 12480V in SATIP
2014-12-11 22:59:31.503 [ TRACE]:satip: SAT>IP DVB-S Tuner #8 (10.0.0.71) - waiting for dvr thread
2014-12-11 22:59:31.503 [ TRACE]:satip: SAT>IP DVB-S Tuner #8 (10.0.0.71) - input thread received shutdown

And my ping/wget (done every minute via cron) gave:
11.12.2014 22:59:01 ping_iplnb.py 1228 (tvheadend is running) returncode=0 wget
11.12.2014 23:00:01 ping_iplnb.py 1244 (tvheadend is running) returncode=0 ping
11.12.2014 23:00:04 ping_iplnb.py 1244 (tvheadend is running) returncode=4 wget
11.12.2014 23:01:01 ping_iplnb.py 1262 (tvheadend is running) returncode=0 ping

So, wget worked 22:59:01h, and failed 23:00:04. Last trace was 22:59:31. What does it mean?

I guess, nothing. If the log is written AFTER the action, and this action fails and result in a hang, then we will never get a log about the last thing responsible for the hang.

I have now restarted everything, diabled all tuners except the first, and did all settings as ckarrie suggested (was like his anyway, except for tune delay, which had been 50 in my case).

[perexg]

The log is incomplete and the control communication (RTSP) is missing (--trace httpc).. httpc means http client (the RTSP client shares same code). There should be also status lines (RTCP) - it might give an idea, if the box hangs on the SHUTDOWN command.. Anyway, I would suggest to create new issue - https://tvheadend.org/projects/tvheadend/issues and upload logs there. I'll check.

[perexg]

Note I moved all SAT>IP code the separate thead in current master, so when one SAT>IP device hangs, other should work.

Also, the second RTSP SHUTDOWN command must be enabled manually now - perhaps, it may cause the hangs for your LNB box.

[ckarrie]

@perexg: Update still works fine for me. @ullix any improvements at your site?

[ullix]

I was travelling the last days, so had to restart the tests. So far it is running ok for a couple of hours with 3.9.2252g1145484precise

At least I found the reason why httpc hat not been traced, although I did enter it: the parser failed to remove a space character. Reported as bug #2552 (https://tvheadend.org/issues/2552).

[ullix]

It hung again ...

I made a bug report (https://tvheadend.org/issues/2555) and added the log.

@perexg What do you mean with "the second RTSP SHUTDOWN command must be enabled manually now - perhaps, it may cause the hangs for your LNB box." ?

Is that something I need to do? Where and how?

[perexg]

Little note for gssbox (and compatible boxes - IDL4K firmware) owners:

There is a new workaround in the latest tvh code which should fix ocassional RTSP 404 errors (bad tunes). I was able to hit this error when I set idle scan for 3 tuners. Basically, the firmware sometimes closed the RTSP connection with 404 error code after new SETUP because previous session was alive (even after TEARDOWN), so another - extra TEARDOWN command must be send to device with previous session and stream IDs to close the old (obsolete) streaming.

EDIT: All tests were done with default SAT>IP tuner settings in tvh.

[clandmeter]

@perexg there seems to be a newer firmware according to http://www.vdr-portal.de/board19-verschiedenes/board52-andere-hardware/p1227048-howto-idl4k-basierte-boxen-modifizieren-digibit-r1-dsi-400-idl400s/#post1227048
Have you tried it?

[perexg]

@clandmeter : According this jollyjinx/idl4k@6049416 (yes, some parts of firmware are now available in sources), only some multicast streaming issues were fixed in V1.17.0.120 . But I'll try.

[clandmeter]

I think i also had issues with my network, and i see a patch for the realtek nic. I will also give it a try.

[perexg]

The 404 error is also in V1.17.0.120:

[   INFO] subscription: 0141: "scan" subscribing to mux, weight: 1, adapter: "SAT>IP DVB-S Tuner #1 (192.168.100.236)", network: "DVB-S 1W", mux: "11843V"
[WARNING] satip: SAT>IP DVB-S Tuner #1 (192.168.100.236) - RTSP 404 ERROR (retrying)
[   INFO] satip: extra shutdown done for session D51C71EE