Docker support

[drouarb]

Create a docker container to bundle the transcoder and allow an easy setup

[drouarb]

Working on it

[Starbix]

Oh I just saw this now, I've also started working on Dockerfiles

[drouarb]

Ok I don't have a lot of time, I can't do it soon

[davidjameshowell]

I'm working on my own solution as well.

I do have a general question though - the biggest underlying issue I'm seeing is with the codecs. I do not know if we can redistribute these (and I'm inclined to say no due to distribution rules).

Additionally, it seems these codecs appear as needed. In a current server of mine, I have ~10 libraries. However in a newly created server (which requires it to be setup and at least played some content) I found only one (base h264 decoder library). Does anyone have more insight on how these are generated or the licenses attached to them?

[Maxou44]

I think Plex downloads codecs when you stream a format for the first time, it could be interesting to check how they did it.

[Starbix]

I solved it this way: https://github.com/Starbix/dockerimages/blob/master/unicorn-transcoder/transcoder/Dockerfile#L60
Which I found here: https://github.com/kayrus/plex/blob/master/README.md

liblibhevc_encoder however throws a 404.

[Starbix]

I have no idea about licensing however, maybe we need to download them at runtime. The same thing may apply to EasyAudioEncoder.

Another problem I've encountered is with the ffmpeg image. If the config is baked into the binary, we can't compile it into the Dockerfile, which mean we'd need to compile it at runtime which is suboptimal.

[drouarb]

We can distribute the codecs and FFMPEG it is a GPL project. We can also compile it from source but Plex team don't release the source code at every update of PMS. I don't know the license of EasyAudioEncoder, we should ask the Plex team if we can distribute it. The version of EAE and FFMPEG is related to the version of PMS.

[b-rockx]

Is it possible to release the dockerfile for this? Users can just copy w/e file is needed until the licensing is worked out.

[Starbix]

I have experimental images however I haven't tested them and I currently don't have time to write documentation or offer support.

[b-rockx]

Thanks. I'll take a look.

[drouarb]

If it can helps: here is the list of codecs needed

libaac_decoder.so
libaac_encoder.so
libac3_decoder.so
libac3_encoder.so
libdca_decoder.so
libflv_decoder.so
libh264_decoder.so
libhevc_decoder.so
liblibmp3lame_encoder.so
liblibx264_encoder.so
libmp2_decoder.so
libmp3_decoder.so
libmpeg2video_decoder.so
libmpeg4_decoder.so
libmsmpeg4v3_decoder.so
libpcm_bluray_decoder.so
libvc1_decoder.so
libvp8_decoder.so
libvp9_decoder.so
libwmav2_decoder.so
libwmv2_decoder.so

Also here is a gist to build Plex Transcoder from source with the codecs, it may be outdated or maybe some codecs are missing
https://gist.github.com/drouarb/fb082c521d46aa43fdbb8cdc3d61ffbc

I tried a long time ago to run a UnicornTranscoder on an ARM server, but the optimization of libx264 on ARM is a crap.

Also I don't think Plex Transcoder can encode in HEVC but I'm not sure

[DrEVILish]

Why not give people a scripted way to extract the Licenced files from the PLEX download source at the containers first run time.

The container image becomes even smaller because it's just a set of instruction to perform an installation at run instead of a single large docker container.

[DrEVILish]

Creating a Docker Swarm for this setup would streamline the setup and addition of new nodes much faster into the transcoding cluster.

Docker service names:
NRP ( nGinx Reverse Proxy )
ULB ( UnicornLoadBalancer )
UTR ( UnicornTranscoder ) as many replicas as your swarm can handle
PLX ( Plex with UnicornFFMPEG )
RED ( Redis KVS )

The ability to use a Nginx Reverse Proxy using the Docker Service Discovery DNS names of the services, would allow a request path:
CLIENT --> NRP --> ULB --> 302 (UTR-N) --> UTR-N --> PLX --> RED --> UTR-N --> NRP --> CLIENT

ULB can nslookup tasks.ULB and would find all of the running ULB containers and return their names and IP addresses. It then performs it's load balancing and returns the 302 to UTR-N

https://docs.docker.com/docker-cloud/apps/service-links/#discovering-containers-on-the-same-service-or-stack

This means that the 302 command doesn't reach the end user only to the NRP allows your network to have a single endpoint for the service.

When a new UTR container comes up it can announce to the KVS that it's arrived, which would trigger ULB to update its cluster info.

[DrEVILish]

Another problem I've encountered is with the ffmpeg image. If the config is baked into the binary, we can't compile it into the Dockerfile, which mean we'd need to compile it at runtime which is suboptimal.

Why not at runtime copy the plex libs from the PLEX host onto the transcoders.
Create a shared folder anywhere on your network that all the docker nodes can access /share/plex/libs/etc
On Start PLEX host cp /local/plex/libs /share/plex/libs
Copying the files ensures that you're not relying on network files to start your transcode.
No need to build or reconfigure.

One issue with this approach is that each of the Docker Nodes would have to Match the Arch of the PLEX Host.

[Maxou44]

Another problem I've encountered is with the ffmpeg image. If the config is baked into the binary, we can't compile it into the Dockerfile, which mean we'd need to compile it at runtime which is suboptimal.

We are working on the 2.0, the UnicornFFMPEG will directly talk with our UnicornLoadBalancer :)

[drouarb]

This means that the 302 command doesn't reach the end user only to the NRP allows your network to have a single endpoint for the service.

The problem of doing that, is that the Nginx become a bottleneck, it's better to 302 to have a direct route to the server.

[DrEVILish]

The problem of doing that, is that the Nginx become a bottleneck, it's better to 302 to have a direct route to the server.

This means that you would end up have multiple endpoints to expose publically, you still would have nginx as a bottleneck when applying the SSL certificate.
nginx would easily handle the traffic of the stream through the network, it has the full support of being itself a load balancer.
If it was Apache I would agree but nginx even running on a raspberry pi I'd be surprised if 4 1080p transcode streams become bottlenecked by nginx.

[drouarb]

It's not Nginx the problem, but the bandwidth of the host server, if you have a 100Mbps server, you will not be able to serve more than 4 1080p streams. For the certificate, I think a let's encrypt wildcard would be fine. I still need to make some research about it.

[DrEVILish]

Okay I see what you're doing you're exposing multiple external domains. I was more thinking for mostly Internal use, with a max of 3 streams going external at once.

[drouarb]

3 streams can be easily handled by a legacy Plex Media Server and a cheap server. In France for 20€/month you can have a hosted PMS that is able to handle 8 streams.

[magn2o]

I have made available a suite of Docker containers that will run alongside an unmodified Plex container. I have been using these containers in production for weeks without issue. The transcoding container will query the designated Plex Server for its current version and then gather the corresponding codec and EAE versions automatically at startup -- the user will no longer need to modify config files or environment variables. I have been using this with my PlexPass servers with are set to auto-update.

The containers are available on GitHub and DockerHub.

[luisfavila]

@magn2o How do you handle plex auto-updates with your solution? Do you manually restart the unicorn-transcoder containers?

[eriksneff]

I'm a little late to the party here, but in response to @luisfavila's question about handling plex updates, according to @magn2o's unofficial UnicornTranscoder docker image: "One of the unique features of this container is that it will automagically identify the Plex version and necessary codec builds at runtime. This means that you can update your Plex server at will and then simply restart this container to maintain version parity without any manual edits."

[Xarin]

I think getting a docker setup for this project would be a huge help for the community testing/using it, and maybe reduce on the amount of support threads for people trying to get it up and running. @magn2o has a great start, I just set it up and got everything working. The only bug I have is the stream stops when (according to tautulli) the transcoder reaches 100% progress. Would it be possible to pull this into the official repo, and maybe start getting some more eyes on the code to support it? If so I could throw up some documentation/templates on my setup.

[Maxou44]

I talked with @drouarb in the past and we are a bit disagree of how magn2o dockers works, for us the best way to do it correctly is to only publish 2 dockers:

• The first one with Plex, UnicornTranscoder, UnicornFFmpeg, and patched stuff (Plex relay, port access, dlna,...)
• The second one with the Unicorn Transcoder

We will not support dockers if the load balancer isn't bundle with Plex because it's really complicated to setup, and it make it harder to setup for the community than classic install 😔

I'm not an expert with Docker, if someone can take the original Plex docker and patch it, it can be really nice, and it will be a pleasure for us to support and update these Dockerfile 😉

[davidjameshowell]

I talked with @drouarb in the past and we are a bit disagree of how magn2o dockers works, for us the best way to do it correctly is to only publish 2 dockers:

• The first one with Plex, UnicornTranscoder, UnicornFFmpeg, and patched stuff (Plex relay, port access, dlna,...)
• The second one with the Unicorn Transcoder

We will not support dockers if the load balancer isn't bundle with Plex because it's really complicated to setup, and it make it harder to setup for the community than classic install 😔

I'm not an expert with Docker, if someone can take the original Plex docker and patch it, it can be really nice, and it will be a pleasure for us to support and update these Dockerfile 😉

So I actually went a secondary approach. I have this successfully broken out into 3 components:

• Plex Master
• Plex Load Balancer
• Plex Transcoder

Everything works perfectly fine!

With the Plex Master, I build a base Plex image (off of LinuxServer) with my specific community tweaks. Then I have another pipeline which builds the Unicorn Base Image which builds the ffmpeg-to-redis binary, replaces it , and then removes the Plex Relay executable.

The load balancer is default and the transcoder are the same, default build with my tweaks.

You just need to ensure that the Plex Master and Load Balancer are on an internal network so they can reference each other of 32400. Bundling the load balancer was not necessary for me and keeps things a bit easier.

I haven't cleaned up any of my configs since they're pretty custom to my nature and being completely Cloud storage based for media (which assists greatly in the transcoder capabilities). I also do hard version locking to prevent version upgrades outside of my own.

[Xarin]

@Maxou44 I do see how bundling the load balancer with plex could reduce some complexity but cant you achieve the same thing with a docker-compose.yml file? This would also allow you to test the load balancer against different versions of plex, and could prevent extra work in the long run? Rather then having to build a new unicorn_lb/plex container every release you would only need to rebuild if a bug was introduced. Apologies if this has already been discussed.

[Maxou44]

We test software without docker and we don't plan to support multiple plex versions, we only recommend to use the latest supported build (check releases notes). It's a bit hard to patch source code using docker, it could make edit/patch harder (for us). Keep in mind it's a really basic node app, a simple npm i && npm start will init and launch transcoder/load balancer 😉

A docker-compose isn't the best solution for majority of people, a lot of people try to setup the project as plex beginners, and it's hard for us to help them. There are too many tweaks to do (32400 not exposed, Plex relay, DLNA, fake binary, disable hardware transcode). The goal of a docker file is to make it "plug and play" 🥰

The main goal of using the same docker for Plex and UnicornLoadBalancer is to limit the risks of misconfiguration, in general if you have a custom plex setup you have the skills to setup the transcoder/load balancer without docker 😎

I can provide dev/debug dockerfile if needed, but it will be useless for majority of people

[Xarin]

Thats fair, in that case this project might be a good candidate for the linuxserver/plex "docker mods" repo but I haven't had a chance to mess with that feature yet.

[davidjameshowell]

Thats fair, in that case this project might be a good candidate for the linuxserver/plex "docker mods" repo but I haven't had a chance to mess with that feature yet.

Let me play around tonight and get a Docker mod working; that'd be really neat.

This is of course assuming that LinuxServer has the version that the Unicorn team has upgraded the configs to (which has been not the case one time before).

[davidjameshowell]

https://hub.docker.com/r/nowaidavid/plex-unicorn-transcoder
https://github.com/davidjameshowell/djh-projects-lsio-docker-mods/blob/master/plex-unicorn-transcoder/Dockerfile

I haven't cleaned this up or for that matter even tested it (as I don't have a viable Plex testing installation currently available). This is mostly a port of my current setup but built in as a Docker Mod. I'd expect this to work, but cannot be for certain given where I am doing replacements (within the cont-init.d).

Add this to a LS.IO Plex Docker image supported by the transcoder (do not set version in environment vars to prevent updating). You will need to ensure that all env vars from the ffmpeg and load balancer are present! If anyone tests this, please comment on the Github with some logs and such.

YMMV with this and is considered HIGHLY experimental and for users familiar with Docker Mods, the Unicorn Architecture, and debugging via Docker.

[magn2o]

I talked with @drouarb in the past and we are a bit disagree of how magn2o dockers works, for us the best way to do it correctly is to only publish 2 dockers:

• The first one with Plex, UnicornTranscoder, UnicornFFmpeg, and patched stuff (Plex relay, port access, dlna,...)
• The second one with the Unicorn Transcoder

We will not support dockers if the load balancer isn't bundle with Plex because it's really complicated to setup, and it make it harder to setup for the community than classic install 😔

I'm not an expert with Docker, if someone can take the original Plex docker and patch it, it can be really nice, and it will be a pleasure for us to support and update these Dockerfile 😉

GASP You wound me, sir! lol

Fair enough, I went ahead and put something together per your scope. This uses the official Plex Docker as a base and then throws the UnicornLoadBalancer and UnicornFFMPEG on top of it. It's self-updating, like the original Plex Docker, where whenever the container is started it will reach out and check for new releases.

Available here: https://github.com/magn2o/UnicornDockerUnofficial

[Xarin]

I got @magn2o 's new build up and working and streamed a 24 min show without issues!

[magn2o]

Added a new env variable (APPLY_IPTABLES) which when set to true will apply rules to drop WAN traffic to 32400 while still allowing LAN traffic. This should let users run host networking (--net=host) without having to try and figure out how to block 32400 traffic. Additionally, this does require the --cap-add=NET_ADMIN flag to be set.

[Maxou44]

@magn2o Maybe we can open a UnicornTranscoder/Docker repository and you could maintain it directly on it? 😄

We can invite you to the organization 😋

[Maxou44]

Do you agree @magn2o ? :)

[Troubadoure]

I got @magn2o 's new build up and working and streamed a 24 min show without issues!

Could you share how you got this running?

[Xarin]

@Troubadoure you'll need to build the image using docker build, and then bring the image up either with docker-compose or docker run. I'll list my setup below, some of the configs may be extraneous since I based them off his original image, and includes some unique volume setups to avoid issues with running plex on an nfs mount.

docker build --tag unicorndocker:2.0 .

docker-compose.yml

---
version: "3"
services:
  plex:
    image: unicorndocker:2.0
    networks:
      - plex-network
    ports:
      - 32400:32400/tcp
      - 3005:3005/tcp
      - 8324:8324/tcp
      - 32469:32469/tcp
      - 1900:1900/udp
      - 32410:32410/udp
      - 32412:32412/udp
      - 32413:32413/udp
      - 32414:32414/udp
      - 3001:3001/tcp
    environment:
      - PLEX_UID=1000
      - PLEX_GID=1000
      - TZ=US/EASTERN
      - PLEX_CLAIM=
      - PLEX_PATH_SESSIONS=/config/Library/Application Support/Plex Media Server/Cache/Transcode/Sessions
      - DATABASE_SQLITE_PATH=/config/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db
      - REDIS_HOST=redis
      - REDIS_PORT=6379
      - SERVER_PUBLIC=https://plex.domain.com/
      - SERVER_PORT=3001
    volumes:
      - /docker/volumes/plex.domain.com/config:/config
      - /etc/plex/local:/config/Library/Application Support/Plex Media Server/Plug-in Support
      - /mnt/Media:/mnt/Media
    restart: unless-stopped
  redis:
    image: redis:alpine
    networks:
      - plex-network
    sysctls:
      - net.core.somaxconn=511
    restart: unless-stopped
  tautulli:
    image: linuxserver/tautulli
    networks:
      - plex-network
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=US/EASTERN
    volumes:
      - /etc/tautulli:/config
      - /docker/volumes/plex.domain.com/config/Library/Application Support/Plex Media Server/Logs:/logs
    ports:
      - 8181:8181
    restart: unless-stopped

networks:
  plex-network:

For the transcoder I just built it the traditional way on another host.