docker-entrypoint.sh overwrites .conf settings, sometimes incorrectly

[jicksta]

We've had a pretty interesting day tracking down why our almost-ready-to-go-live Neo4j instance was running out of memory on a 32GB machine with a 10GB page size set in our neo4j.conf. We discovered that the neo4j-wrapper.conf running in the container had changed at container runtime and had different memory settings than what were baked into the image's config files.

The docker-entrypoint.sh script invariably overwrites settings in the config files based on (undocumented?) environment variables, for example:

• $NEO4J_dbms_memory_heap_maxSize
• $NEO4J_dbms_memory_pagecache_size
• et al...

This was really surprising to us and hard to track down. We were using COPY commands in our FROM neo4j:enterprise Dockerfile to copy our customized .conf files into our derivative image, but these files' settings were being overwritten without any obvious cause (until we looked in the ENTRYPOINT).

Our workaround was to define these environment variables in our Dockerfile:

# NEO4J_dbms_memory_heap_maxSize is in megabytes
ENV NEO4J_dbms_memory_heap_maxSize   10000

# NEO4J_dbms_memory_pagecache_size is in kilobytes
ENV NEO4J_dbms_memory_pagecache_size 9000000

Furthermore, we noticed that dbms.memory.heap.initial_size setting was being set to the value intended for dbms.memory.heap.max_size, and that there is no way to set the initial size via these variables. Even if we set the setting's value in our config file, this entrypoint would just overwrite it. We'd probably need to COPY in our own customized /docker-entrypoint.sh in to get around this bug.

I noticed that the two separate config files (neo4j.conf and neo4j-wrapper.conf were being merged together in Neo4j 3.1 but maybe along with 3.1 we could figure out a better way to manage these config settings for the Docker image?

It seems standard practice across Dockerized services to respect config files first and then overwrite a setting iff an environment variable was actually defined. Modifying the config file in the container at container initialization time is probably not the most elegant option for making the environment variable overrides, either.

Thoughts?

[benbc]

@jicksta I'm sorry that you've had trouble with this behaviour. I can see that it's surprising if you are deriving your own image from the standard one and copying in config files -- we hadn't considered that case when designing the configuration mechanism.

The intended way of providing your own complete config files to the image is via a /conf volume. If you were to add config files to /conf in your derived image, you would find that they don't get modified and you won't need to define the environment variables.

Neo4j itself does not have any way of injecting configuration via environment variables, so the only way that the Docker image can provide this is by modifying the config files. We hope to enhance Neo4j so that it can take config from the environment natively, but we won't be able to do that for 3.1.

The environment variables that the image responds to are documented here. You'll see that the only ones we provide default overrides for are the memory-related settings. This is because Neo4j's default behaviour is tuned for a server deployment where Neo4j is the only resource-hungry software running on the server; so by default Neo4j will consume the majority of the available memory. We consider that to be inappropriate for container deployment, so we've limited the default memory usage.

On the specific question of initial vs maximum heap size, our standard recommendation for Neo4j is to configure them to be the same. This gives much more predictable memory consumption, avoiding problems where servers run out of memory under load due to individual queries. We decided to bake this recommendation into the Docker image to simplify things for operators.

[jicksta]

@benbc

The intended way of providing your own complete config files to the image is via a /conf volume. If you were to add config files to /conf in your derived image, you would find that they don't get modified and you won't need to define the environment variables.

I'll give this a try. I should add tho that AFAIK it's pretty common to COPY config files into a derivative image. Volumes can bring their own hassles, so I'm not sure why having a volume for just /conf would be helpful to our particular use case.

Also thanks for the link the to documentation about the env variables.

On the specific question of initial vs maximum heap size, our standard recommendation for Neo4j is to configure them to be the same. This gives much more predictable memory consumption, avoiding problems where servers run out of memory under load due to individual queries. We decided to bake this recommendation into the Docker image to simplify things for operators.

Consider that most folks using dockerized Neo4j in production would probably have Neo4j wired up into a local Docker-Compose (or similar) system running on a local development environment. We want to set the initial memory size to be something smaller so we can have a Neo4j instance running in the local devmode cluster that doesn't suck up precious laptop memory resources unless necessary.

[benbc]

@jicksta The /conf volume is intended for direct users of the standard Neo4j image, not for people extending the image. I only mention it because it gives you a simpler workaround for your problem.

[benbc]

@jicksta I take your point about Compose etc. However we think that simplifying the surface and removing the possibility for nasty surprises in production is more important in this case.

[adamrothman]

@benbc I am still having trouble with this, despite using a /conf volume as directed. This is how I'm running Neo4j:

docker run -d \
    --publish 7474:7474 \
    --publish 7687:7687 \
    --restart=always \
    --volume /ebs/xvdi/conf:/conf \
    --volume /ebs/xvdi/data:/data \
    --volume /ebs/xvdi/logs:/logs \
    neo4j

In /ebs/xvdi/conf/neo4j.conf I have uncommented and modified these lines:

dbms.memory.heap.initial_size=1G
dbms.memory.heap.max_size=1G

But Neo4j is only running with 512M, judging by the output of docker top:

$ sudo docker top eager_swartz
UID                 PID                 PPID                C                   STIME               TTY                 TIME                CMD
root                1731                1714                1                   Oct24               ?                   00:00:26            /usr/lib/jvm/java-1.8-openjdk/jre/bin/java -cp /var/lib/neo4j/plugins:/var/lib/neo4j/conf:/var/lib/neo4j/lib/*:/var/lib/neo4j/plugins/* -server -Xms512M -Xmx512M -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+AlwaysPreTouch -XX:+UnlockExperimentalVMOptions -XX:+TrustFinalNonStaticFields -XX:+DisableExplicitGC -Djdk.tls.ephemeralDHKeySize=2048 -Dunsupported.dbms.udc.source=tarball -Dfile.encoding=UTF-8 org.neo4j.server.CommunityEntryPoint --home-dir=/var/lib/neo4j --config-dir=/var/lib/neo4j/conf

This is with the official 3.2.5 CE image.

[adamrothman]

I did some more digging in my running container.

It looks like docker-entrypoint.sh makes a copy of /conf/neo4j.conf and overwrites my un-commented lines.

docker-neo4j/src/3.2/docker-entrypoint.sh

Lines 93 to 106 in a846084

	# list env variables with prefix NEO4J_ and create settings from them
	unset NEO4J_AUTH NEO4J_SHA256 NEO4J_TARBALL
	for i in $( set | grep ^NEO4J_ | awk -F'=' '{print $1}' | sort -rn ); do
	setting=$(echo ${i} | sed 's|^NEO4J_||' | sed 's|_|.|g' | sed 's|\.\.|_|g')
	value=$(echo ${!i})
	if [[ -n ${value} ]]; then
	if grep -q -F "${setting}=" conf/neo4j.conf; then
	# Remove any lines containing the setting already
	sed --in-place "/${setting}=.*/d" conf/neo4j.conf
	fi
	# Then always append setting to file
	echo "${setting}=${value}" >> conf/neo4j.conf
	fi
	done

This directly contradicts the documentation, which states:

Any configuration files in the /conf volume will override files provided by the image. This includes values that may have been set in response to environment variables passed to the container by Docker.

Please advise.

[adamrothman]

@spacecowboy @srbaker 👆 (sorry, I'm not sure who the right people are to ping)

[benbc]

Sorry you're having trouble with this @adamrothman. We'll take a look and get back to you.

[sonu27]

Any update on this?

[adamrothman]

@sonu27 in the mean time, I have been working around this issue by setting the NEO4J_dbms_memory_heap_initial__size and NEO4J_dbms_memory_heap_max__size environment variables in my docker run command:

docker run -d \
  --env NEO4J_dbms_allow__format__migration=true \
  --env NEO4J_dbms_memory_heap_initial__size=1G \
  --env NEO4J_dbms_memory_heap_max__size=1G \
  --env NEO4J_dbms_security_auth__enabled=false \
  --name neo4j \
  --publish 7474:7474 \
  --publish 7687:7687 \
  --restart=always \
  --volume /ebs/xvdi/data:/data \
  --volume /ebs/xvdi/logs:/logs \
  neo4j:3

[praveenag]

Hi @adamrothman Apologies that it took a while to reply to you. We think the documentation is at fault here. It should instead say

Any configuration files in the /conf volume will override files provided by the image. Environment variables passed to the container by Docker will still override the values in configuration files in /conf
volume

With the updated documentation, your work around will be the correct way to achieve what you want.

[praveenag]

Updated instructions can be viewed here

https://neo4j.com/docs/operations-manual/current/installation/docker/#docker-conf-volume