[Q] How to log rotate go-carbon log file in Docker container?

[flucrezia]

stderr cannot be accessed with a non-root user in Docker containers as discussed here, so how can log rotation be applied to an arbitrary log file?
Should I relay on the host log rotation system by bind mounting the file?

Many Thanks,
Francesco

[bom-d-van]

Should I relay on the host log rotation system by bind mounting the file?

hi yeah, I think using logrotate would be a way to go. that's what we are using on prod.

(also the lib zapwriter used in go-carbon detects log rotation and it will re-open log files after rotation).

[flucrezia]

Hi, many thanks for the reply.
Just one question, are you using the user carbon in the Docker container or the root user?
I used to modify the Dockerfile to run the root user in the past. Now that I'm trying with the non-root user carbon, without modification to the Dockerfile, I cannot make it work. I get a permission denied either with stderr or an arbitrary filename for logging. I tried to bind the directory /var/log/go-carbon/ to an host directory and set UID and GID of the carbon user as those of the directory owner (non-root user with id 1000). But it's always permission denied...

[bom-d-van]

Just one question, are you using the user carbon in the Docker container or the root user?

we run go-carbon on bare metal servers with carbon as user. by your description, it sounds like generic docker setup issue.

maybe @deniszh has some insight?

[deniszh]

We're using carbon in Docker image - https://github.com/go-graphite/go-carbon/blob/master/Dockerfile#L22
For the rest - I can't help here, sorry. Some issue with permissions, but looks like related to Docker and your system. As @bom-d-van said we're not using that docker container in production.

[flucrezia]

No problems. I'll make some other tests in order to have a clear view of what is happening.
Currently, I can say that with the root user, and so commenting out USER carbon in the Dockerfile, using "stderr" as logger output, it works smoothly, while using a non-root user or specifying a filename as logging output it does not.
Many Thanks.

[deniszh]

If you find generic solution - we would like to accept it. Running process as root in containers is not recommended and discouraged by security reasons IIRC, so, that's not generic solution, alas. Thanks for looking into this, much appreciated!

[flucrezia]

Okay, I found two sources of problems:

1. mkdir -p /var/lib/graphite/{whisper,dump,tagging} does not expand correctly in the Dockerfile since the RUN command by default use sh instead of bash. So either we could list full path for each:
   mkdir -p /var/lib/graphite/whisper /var/lib/graphite/dump /var/lib/graphite/tagging
   or change the default shell with:
   RUN [ "/bin/bash", "-c", "mkdir -p /var/lib/graphite/{whisper,dump,tagging}]
   (still to check if && is possible inside the square brackets to avoid adding one layer to the image)

2. My fault using bind mount instead of Docker managed volumes. Everything works fine, with user carbon, using Docker volume in place of bind mount a directory of the host. It is still hard to logrotate an arbitrary file within the container but it should be better to use stdout/stderr as logging output using Docker. Maybe a warning to avoid bind mounts in the README could be useful.