Docker container overhaul

[feederbox826]

Is your feature request related to a problem? Please describe.

• Running as root in the docker container can cause issues with permissions
  • also is a minor security concern Non-privileged user in Docker build #684
  • while USER is an option, it breaks since stash looks under $HOME/.stash for config files
  • since it is not container-wide, ffmpeg might possibly be running as another user? [Bug Report] Inconsistent file permissions in the "generated" directory #2797
• re-installing new python libraries on every new update is very troublesome

Describe the solution you'd like

• A proper LSCR.io - like base for handling common environment variables like
  • TZ Adjustments to time zone settings for Docker #1238
  • PUID/ PGID Non-privileged user in Docker build #684
• Github Container Registry + DockerHub builds [Bug Report] migrate away from dockerhub? #3558
• a proper DockerHub README
• officially supported CUDA/ QSV images (eventually ™️) (Github action to publish a CUDA build alongside the alpine one #4091 related)
• an entrypoint script to install and update python dependencies specified by the user on startup
  • (also resolves closes Add pystashlib to Dockerfile #4102)
• a persistent volume for python dependencies
• a default /config directory instead of /root/.stash

Describe alternatives you've considered
N/A

Additional context
I want to work on this. Please assign me to this. I've done this for other projects and I am eager to finally do it for stash

[feederbox826]

High level overview and TODO tracker

developing in https://github.com/feederbox826/stash-s6

container

• entrypoint *ash script
  • runtime python dep installation (r: Add pystashlib to Dockerfile #4102, [Bug Report] Python Libraries Missing When Running Docker Container As Non-Root User #4677)
    • automatic requirement finding, merging and upgrading
  • migrations/ data storage
    • stashapp/stash
    • hotio/stash
    • custom configurations
    • /config mount
    • mount detection
• rootless/ USER handling (r: Non-privileged user in Docker build #684)
  • rootless mode
  • k8s "security compatibility"
  • permission checks & warnings
  • permission corrections (r: [Bug Report] Inconsistent file permissions in the "generated" directory #2797)
• root container restrictions
  • SKIP_CHOWN flag
  • PUID/ PGID user switching
• misc
  • TZ ENV support (r: Adjustments to time zone settings for Docker #1238)
  • arm <v7 support
• hardware acceleration
  • ffmpeg flavors
    • jellyfin-ffmpeg
    • lscr.io build
    • stock (debian)
  • hardware acceleration tests (r: Github action to publish a CUDA build alongside the alpine one #4091)
    • lscr.io without drivers removed (hwaccel changes and LSCR.io removal feederbox826/stash-s6#2)
    • base ffmpeg driver installs (across arch)
    • jellyfin without drivers
    • VAAPI iHD (gen8+)
    • VAAPI i965 (gen5+)
    • QSV (Intel)
    • Intel ARC
    • NVENC (NVIDIA)
      • multi-stream NVENC
    • V4L2 (rPi)

Building

• Makefile
• Github Actions
  • simultaneous arch builds
  • Builds from source
• publish to docker.io, ghcr.io (r: [Bug Report] migrate away from dockerhub? #3558)

Documentation

• updates
  • update dockerhub README
  • add links to hwaccel flavors, make images links
  • update docker-compose ENV, mounts, GPU bindings
• additions
  • docker image
    • add documentation for entrypoint variables
  • hw acceleration
    • breakdown of flavors/ hardware acceleration
    • link test videos for reencoding
    • example arguments
    • device mounts

[Leopere]

The theory of why the changes need made is on par with what I’d want implemented. Privilege reduction proper installation of software requirements my biggest issue is that it needs to help users from before keep working ideally

[feederbox826]

it needs to help users from before keep working ideally

Yep, my plan right now is to test it in my own repository to iron out any bugs, and write mgirations for both hotio/stash and stashapp/stash so that users can migrate over seamlessly or as seamless as mv && chown -R can get

[stg-annon]

another problem people have with docker installs is python packages, might be worth taking a look at setting up a venv so that plugins and scrapers that require other packages don't need to reinstall them on every reboot

alternately we may be able to get away with just setting a couple environment variables to a mount from the host
PIP_TARGET
PYTHONHOME

[maverickd650]

Could we keep a rootless container option without s6? It causes particular problems on kubernetes as it aggressively CHOWNs files and can cause a container to fail to start by exceeding the probes time limits. It must also be run as root on kubernetes to allow the inital s6 operations to take place which isn't ideal since we're aiming for rootless.

[feederbox826]

alternately we may be able to get away with just setting a couple environment variables to a mount from the host
PIP_TARGET
PYTHONHOME

https://github.com/feederbox826/stash-s6/blob/b71eb246ae412192e1d9426cc01b4829c546738f/alpine.Dockerfile#L16-L19

cache and PIP_INSTALL_TARGET are both set to a volume

[feederbox826]

Could we keep a rootless container option without s6? It causes particular problems on kubernetes as it aggressively CHOWNs files and can cause a container to fail to start by exceeding the probes time limits. It must also be run as root on kubernetes to allow the inital s6 operations to take place which isn't ideal since we're aiming for rootless.

the chown operations are done by the init scripts, not by s6 itself. Unfortunately my goal with this is to be as beginner-friendly as possible, following lscr.io conventions and s6 allows us some leniency without creating a excessively long docker-entrypoint script and adding in modular dropins for different flavors. I'll look into creating a minimal (alpine + python + binary) image, but I'm not a fan of fragmeting further from the base

[Crow-Control]

A proper LSCR.io

following lscr.io conventions

From a professional perspective, LSIO containers are not good containers.
They are borderline problematic on anything but a docker(-compose installation), for example on kubernetes. Due to, essentually, requiring root. They essentially cut corners from good container practices, to build a container that "just works" for the average joi.

Calling LSIO a "convention", makes it seem like it's somehow a high grade container standard, which it's quite obviously not.
It's better than the average container, but that comes at a cost.

also is a minor security concern #684

To be clear: The main container still runs as root in S6 and requires multiple(!) layers of security precautions in kubernets to be disabled. It also prevents things like supplemental groups to be used correctly.

while USER is an option, it breaks since stash looks under $HOME/.stash for config files

The solution for which is adapting this behavior, not migrating to S6 overlay based containers

[feederbox826]

Either way I'm discussing with @Leopere to drop s6 in favor of a (b)ash script but,

From a professional perspective, LSIO containers are not good containers.

They're not good but they work out of the box for 99% of users which imo is more important than accomodating someone who is already building their own helmchart and should know how to build their own container

k8s, as much as we would like to try and dance around, is not exactly docker, but that's a tangent for another time.

To be clear: The main container still runs as root in S6 and requires multiple(!) layers of security precautions in kubernets to be disabled. It also prevents things like supplemental groups to be used correctly.

iirc #684 is not to run a root-less container, is to run as a non-priviledged user within a rooted container.

[feederbox826]

I'll try to get the bash scripts out the door, but I have a some personal negative experiences with k8s but PRs are always welcome.

[Crow-Control]

imo is more important than accomodating someone who is already building their own helmchart and should know how to build their own container

Writhing a helm chart is a lot different than writhing containers as they are a LOT easier to standardise and maintain. Easier in terms of dev-time per application. Just because I can technically do something, doesn't mean the project I maintain has the manpower to do so on a scope of 900 help charts.

k8s, as much as we would like to try and dance around, is not exactly docker, but that's a tangent for another time.

Well, in terms of security: Yes it offers a lot more options to lock-down. But in general it has quite clear best-practices that aline with docker itself as well in certain areas:

But it's not rocker science:

• Accept USER being set
• Dont change user
• Don't chown (without copy)
• Dont chmod
• Don't write to directories already containing files (only write to folders that can be mounted)

[scruffynerf]

As someone who has written for and used both k8s and docker, in the context of Stash, k8s makes no sense for most people. Docker is a much more easily supported standard, and more so, the level of support it generates is already high. K8s would not improve matters.
As much as I feel like some of the docker users could be better served by a native app, at least there are reasonable arguments in favor of Dockerizing.

[feederbox826]

Dropped s6 in favor of an ash entrypoint script

Limitations of root-less

• volume mounts are by default not accessible
• nvenc stream needs root to run ldconfig
• cannot migrate from root to non-root

but that's inevitable. If you have further suggestions, you seem much more versed in rootless and k8s compatibility so I look forward to your PRs