Add transcoding via filter scripts to http streaming

[BtbN]

This adds a new parameter to the http streaming server.
Adding ?transcoder=test makes it pipe all http output through a "filter", which can do basicaly anything with it.

It searches for the trascoders in hts_settings_get_root()/transcoders
It also passess all GET parameters in hc_req_args as environment variables, prefixed with GET_, so the transcoder can get options via url parameters.

This also modifies the writing to the socket a bit, so it checks if it can write to the http stream, because it can no longer rely on write() errors, because it no longer directly writes to the tcp socket. I use select and poll for this.

There is still a problem somewhere, because in my tests a transcoding vlc throws a lot of warnings about broken input data. Connecting with vlc via http, which should bring the same data to vlc, works perfectly. But this only seems to happen with higher bandwith streams, so i guess it's some buffer overflowing.

But in general, this already works fine.

[BtbN]

Ok, found why it did not work propperly. VLC uses diffrent caches for file(stdin is seen as file) and network input. And the default 300ms cache-time are not enough.
Here's the transcode script I use for VLC now:

#!/bin/bash

export DISPLAY=:0
exec cvlc --file-caching 1500 "--sout=#transcode{venc=va,threads=4,deinterlace,height=720,vb=-33}:std{access=file,mux=ts,dst=-}" - 2>/tmp/vlc.log

A no-op passthrough-script could look like this:

#!/bin/bash
exec cat <&0

It's working flawlessly for me now!

[pvagner]

Is there a way to somehow be able to signal a proper mime type to the http server? For example I would like to encode audio to aac or vorbis using ffmpeg. Since I am dealing with raw audio proper mime type would be usefull so I can for example play the audio on android phone, symbian phone and even on a windows pc with winamp or foobar 2000.
I am sorry this might sound strange but yes I am only interested in audio and am discarding video by using audioes muxer what is available as a pull request 263.

[BtbN]

No, there is no easy way to influence what headers are sent.
When the transcoder is initialized, the headers have already been sent. Running it earlier is also not possible, because the pipes it needs aren't available.

[pvagner]

This can no longer be applied without editing. There are no functional differences just merge conflicts: http://pastie.org/8137891
Maybe git can do that automagically however I am using this as a patch.

[BtbN]

This already is up-to-date with current master and applies cleanly.

[fritz-fritz]

This seems like a way to get HLS (HTTP LIVE STREAMING) working! if the "filter" can have vlc transcode the stream then we could use vlc's ability to output a live stream that could then be used for native playback on mobile devices (aka iPhone) without the need for VLC to be installed!

I could be wrong... but definitely worth looking into!

[BtbN]

HLS isn't a simple stream like normal http/rtmp Streaming.
It's a playlist which is continously reloaded, and the stream is split into multiple segments.
This patch only supports single continous streams.

[fritz-fritz]

I understand how HLS works. Though it does appear you have the ability to take the live tv stream and feed it into a bash command. VLC has command line arguments to handle live transcoding to HLS.

This is an example of code i've used in the past:

vlc atsc://frequency=677029000 --int=dummy --sout-transcode-audio-sync :live-cache=3000 --sout='#transcode{vcodec=h264,vb=150,fps=25,width=480,scale=1,venc=x264{aud,profile=baseline,level=30,keyint=15,bframes=0,ref=1},acodec=aac,ab=40,channels=2,samplerate=22050}:duplicate{dst=std{mux=ts,dst=-,access=livehttp{seglen=10,delsegs=true,numsegs=10,index=/var/www/stream/live.m3u8,index-url=live-######.ts},mux=ts{use-key-frames},dst=/var/www/stream/live-######.ts,ratecontrol=true}}'

So it seems to me that the HLS stream could at least be generated but as far as impacting how its accessed or how it impacts hts... that I don't know

[BtbN]

I think this would be easier to achive by setting up nginx-rtmp with a hls setup, which is configured to pull from tvh.

[Jalle19]

Some logging would perhaps be appropriate here?

[BtbN]

The mime file is completely optional and it not existing or beeing empty is not an error or even worth a warning.

[Jalle19]

You're right.

[Jalle19]

@BtbN maybe you could drop some of the cosmetic changes (mainly the "opening brace on next line" changes) and move those that fix bad indentation to a separate commit?

[nhsman]

Can you let me know the where to place transcoding script? I'm getting this in TVH log:

2014-04-13 14:03:42.585 webui: Transcoding stream with transcoder /home/vlc/.hts/tvheadend/transcoders/test
2014-04-13 14:03:42.585 webui: Transcode-Script breakout attempt detected

[Jalle19]

@nhsman you'll have to use "test" as the transcoder script, not the full path.

[nhsman]

Thanks. But where to place actual script? I have it in "/home/vlc/.hts/tvheadend/transcoders/" and
when I use just "test" I get this:

2014-04-13 16:15:40.497 webui: Transcoding stream with transcoder test
2014-04-13 16:15:40.497 webui: Transcode-Script "No settings dir/transcoders/test" does not exist or is not executable!

[Jalle19]

Have you made sure the script is executable?

[Jalle19]

And does it really say "dir/transcoders/test"?

[nhsman]

Yes it says "dir/transcoders/test". And yes, the script is executable. Any ideas?

[Jalle19]

Apparently it determines your configuration directory to be "No settings dir", which is why the script fails. How are you running tvheadend and under what user does it run (your posts would indicate the user is "vlc", is that correct)?

[nhsman]

I compiled TVH from source and run it as "sudo tvheadend -C -f".

[Jalle19]

@nhsman don't run it as root.

[nhsman]

If I start tvheadend as non root,
tvheadend -C -f
I cannot see TV Adapters in webif.
My user is member of video group.

[clandmeter]

Mine runs as -u tvheadend and tvheadend is member of the video group.
Make sure your devices have correct ownership set by your device manager.

And remember, if you run tvh as root "ones", to chown -R your config directory (if init doesn't take care of that).

[Jalle19]

@BtbN you should probably update your code to check for NULL after this change: af8e79c , currently http_stream_transcode fails with an incorrect error if no settings directory is present.

[BtbN]

Indeed, i'm not sure how to react in that case. Just abort and assume no script was present?

[nhsman]

Thanks.
I've been able to start TVH like this: "tvheadend -C" and now I can see adapters, but when I try to start it as daemon I got this:

tvserver@SERVERTV:~$ tvheadend -C -f
2014-04-14 22:30:28.681 [ ALERT] START: setgid() failed, do you have permission?

Any ideas?

[Jalle19]

@BtbN: abort with something like "No configuration directory in use, unable to load transcoder scripts"

[Jalle19]

@nhsman something is wrong with your Linux or your tvheadend installation.

[nhsman]

I use Ubuntu 13.04 x64 minimal. I had to upgrade kernel to 3.13 to get my tv adapter works. Then compiled TVH from here: https://github.com/BtbN/tvheadend using "simple_transcode" branch.
As I said previously, after adjusting permissions I can start TVH with "tvheadend -C" and then transcoding works fine but for some reason I cannot start is as a daemon.
Do you think it has something to do with kernel upgrade?

[nhsman]

Hi, I've compiled latest master with this PR and it seems to not work.
When I start stream, eg.
http://192.168.1.70:9981/play/stream/channel/41cb8d1f899b642902a77e4525c4c26e?transcoder=test
it opens stream but according to Tvheadend log it is not even looking for script.
The channel is played as original stream.
Can you please advise. Thanks.

Sorry guys, it's my bad. It should be:
http://192.168.1.70:9981/stream/channel/41cb8d1f899b642902a77e4525c4c26e?transcoder=test
It works fine then.
Thanks.

[Hedda]

You might want to consider borrowing/stealing the UPnP / DNLA client profiles XML files from the latest Plex Media Server and use as a base to create custom MIME types in filter profile per reciever client for transcoding and/or remuxing , see:

https://github.com/Hedda/Plex-DNLA-client-profiles/tree/master

https://github.com/Hedda/Plex-DNLA-client-profiles/archive/master.zip

PS Hope it is OK to have copies of these XML files on GitHub as Plex.tv don't have an source code repository available because rest of the software is closed source.

[perexg]

This functionality was implemented like MPEG-TS Spawn streaming profile where users can configure the command which should be invoked for the original MPEG-TS stream. v4.3-344-gc349525b2, more commits, the main is 5bbddc9 .