Docker image shutting down after a few days

[Mange]

I'm running the service inside the official docker image and proxy it via nginx + let's encrypt. After a few days the process exits with status code 0 and no log output, making nginx shut down (for domain name reasons with docker-compose; but that's not the issue here).

Is there some way I can try to debug this problem? I'm pretty confident in Rust, but it's also running on a server with limited hardware, so recompiling constantly is probably not going to work for me.

It's happened four times now, so I'm now turning on restart: unless-stopped to work around it, but getting to the cause of the issue would be preferred by me.

The little information I can give:

1. Process had no output. Exit status 0.
2. The most I've been able to have the container running at a time: Around 3 days.
3. Service works well when it's running (although it emits a lot of warnings about upgrading websockets.
4. Memory pressure on the server is fine. It was not killed by OOM killer according to docker inspect either.
5. CPU usage has been low up until the service died.
6. No problem with disk space.
7. Server has had no network interruptions during this time.
8. No other container (I run around 11 of them) has any similar issues.
9. Server is my own hardware at home, not a cloud host or a VM. No hardware issues that I know of, and it's pretty new.

I'm sorry I cannot give any more information. This "issue" is literally me asking for ideas on what to look for so I can give you better information.

[izderadicka]

Are you using provided Dockerfile for alpine linux?
What exact config/ rguments are you using to run program?

To get more detailed logging (debug) you'll need to compile in debug mode (without --release, binary then will be in target/debug) and set environment variable to RUST_LOG=debug (which will log debug from all creates) or just RUST_LOG=audioserve=debug - to log only debug messages from this binary.

You can also try to strip down features you are not using - compile with --no-default-features and then --features you want. This will leave out unwanted code in case it causes problems.

When exactly the termination of container happens?

• when streaming some music?
• when working with files in collections directories (copying new, renaming etc)?
• or when audioserve is unused?

As exist status is 0 I assume it's not segfault or some other OS enforced termination.

[izderadicka]

Also looking now at default logging when running in Docker container it was not ideal - although info logging is enabled for release build, it's not enabled by adding appropriate env. variable (I've fixed it in latest master).
So at least I would recommend to have ENV RUST_LOG=audioserve=info in docker file, then you can see if program terminated normally - as this message should be last one:

INFO  audioserve > Server finished

[Mange]

I run it using this docker-compose configuration:

    audioserve:
      image: izderadicka/audioserve
      container_name: audioserve
      command: ["--allow-symlinks", "/audiobooks"]
      volumes:
        - /home/mange/services/audioserve/config:/home/audioserve/.audioserve
        - /home/mange/services/audioserve/library:/audiobooks
      env_file:
        - /home/mange/services/audioserve/config/secrets.env
      restart: unless-stopped

It happens when the service is not used.

I'll see about recompiling it and trying the debug logs for a bit, but now it's been running without issue for 4 days. I'll recompile it the next time it crashes, just before I also capture the whole output of docker inspect so I had additional clues.

[izderadicka]

@Mange Any more crashes? Some logs?

[Mange]

Seems from docker inspect that the current container was started on the 8th of august, a week after my last update. But since then it's been running (Up 2 weeks).

Log is just full of

 ERROR websock > Upgrade is not to websocket
 ERROR websock > It must be upgrade connection

so I can't give you anything from there.

[izderadicka]

Thanks,
the error messages are related to the playback position sharing, which is kind of experimental now (position is sent over websocket as file is playing). If you are not using it you can disable it in client by not specifying group.

[Mange]

I am actually trying to use it, though. 😄

[izderadicka]

@Mange And is it working? Can you open another issue for these error messages? I'm also trying to use it , but getting different messages

ERROR websock                                       > Socket error Ws

I think I know why this message come from - it's when websocket is closed unexpentantly (mobile device lost connection, rev. proxy closes connection). Here it looks like client is using incorrect websocket request - or maybe it's proxy (normally you need to do some setup on rev. proxy for websockets to work).

[izderadicka]

Any update? Still problem with docker image stability?

[Mange]

It seems to have gone away. Feel free to close this. I'll reopen if I see it happening again with any additional information I have. Den fre 27 dec. 2019 17:45Ivan Zderadicka <notifications@github.com> skrev:

…

Any update? Still problem with docker image stability? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#16?email_source=notifications&email_token=AAAAMP5DDLALZHW5JYLTBVTQ2YWLLA5CNFSM4IHKWZ62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHXOLSY#issuecomment-569304523>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAMP6CJR3UIAWCPBGYVJTQ2YWLLANCNFSM4IHKWZ6Q> .

[izderadicka]

@Mange Thanks