Ability to only replace variables in plugin files synced from /data/plugins

[Trikolon]

Since e760407 REPLACE_ENV_VARIABLES is already faster, but it still takes several minutes to do the replace operation on my production servers. That's because there are a lot of files from flatfile storage (e.g. EssentialsX). That means restarts are really slow.

I propose to add a config option to only perform search-replace on the files copied / synced in from /data/plugins etc. This would result in a major speedup for my use case.

My use case: Plugin configs + binaries versioned in a git repository with variables for secrets and other deployment specific configuration settings.

[itzg]

The request makes sense, but it might be difficult to implement. I'll have to give this more thought.

[Trikolon]

I don't have time to work on this myself currently, but I've placed a small bounty on it in case anyone is interested:
https://www.bountysource.com/issues/100086618-ability-to-only-replace-variables-in-plugin-files-synced-from-data-plugins

[stonewareslord]

@Trikolon I would like to take up this bounty. To confirm: this issue is because find is traversing through all of /data/ and you only want it to traverse /data/plugins? My solution below should solve that.

@itzg Does this solution look good to you? stonewareslord@ed4ec1c

It should allow users to pass in multiple paths through wordsplitting and it should not be any less safe (glob wise) than other config variables. If you like this solution, I can document this in the readme and make a PR.

Sample usage: docker run ... -e REPLACE_ENV_VARIABLES_INCLUDE_PATHS="/data/plugins /data/abc" ...

[Trikolon]

Thanks for picking this up!

I don't think the proposed solution would work here. I mount /plugins read only, so the goal would be a mode to replace the config variables before syncing / writing them to /data/, without the need to update /plugins/ in place.
For example, one of my mounts looks like this: ./config/mc-survival/plugins:/plugins:ro.

At the moment /plugins is synced with /data first, before performing the variable replacement on /data/.
We could use pipes when syncing /plugins/ with /data/ to replace the variables while copying files. Alternative we could also add an intermediate step where we create a copy of /plugins in a temporary location, replace the variables and then sync with /data. The latter seems easier to implement.

[itzg]

Oh no, I was afraid this would happen. I didn't want to say anything in case my availability went away but I'm almost finished implementing this. I wasn't doing it for the bounty so I'm glad to hand that off to @stonewareslord for taking the initiative. FWIW I was planning towards having an image later tonight for testing.

[itzg]

@Trikolon there's now an image ready to test at itzg/minecraft-server:test-interpolate-on-copy. It's currently based on the java16 variant, but I can rebuild against java8 if needed.

It uses all the same "REPLACE_ENV" type variables. At this point it always interpolates copied files with suffixes yml, yaml, txt, cfg, conf, properties, hjson, json, tml, toml, so you'll probably want to disable the in-place interpolation by setting REPLACE_ENV_VARIABLES=false.

[itzg]

Updated docs now describes in the latest test image a new pair of variables REPLACE_ENV_DURING_SYNC and REPLACE_ENV_IN_PLACE, Both default to false but REPLACE_ENV_IN_PLACE falls back to the old variable REPLACE_ENV_VARIABLES.

[Trikolon]

@itzg Thanks for working on this! 👏

I did some preliminary testing. Replacing the variables synced from /plugins seems to work. However, when I update the config file in /data to be newer than the one to be synced from /plugins it doesn't overwrite it. PLUGINS_SYNC_UPDATE doesn't seem to apply anymore? The feature is important for me, since plugins sometimes update configurations themselves, which will break syncing my git versioned configs.

The mc-image-helper will still log it, but it won't replace the file. e.g.

mc-test_1  | [mc-image-helper] 11:18:59.424 INFO  : Interpolating /plugins/CoreProtect/config.yml -> /data/plugins/CoreProtect/config.yml

[itzg]

I re-implemented the rsync --update behavior which is actually opposite to what you're desiring. From the manpage:

       -u, --update
              This  forces  rsync  to  skip  any  files which exist on the destination and have a
              modified time that is newer than the source file.  (If an existing destination file

With that said, if it logged "Interpolating" then it is indeed attempting to locate and replace variables in there. Can you provide more specifics about what variables you're expecting replaced, what your REPLACE_ENV settings are, and what those values are in your environment? (Passwords obfuscated of course.)

[itzg]

Actually the correct section that is directly applicable to this feature is here. That's where I have described the new variable REPLACE_ENV_DURING_SYNC and the others used for variable processing.