Recognize Env Var "CONTAINERS_CONFIG_DIR"

[0x0D15]

Would you consider recognizing an environment variable "CONTAINERS_CONFIG_DIR" that overrides the parsing of config files/dirs in other directories (/etc/, /usr/, etc..) like registry.conf, policy.json, and registries.conf.d and forces the exclusive reading of config files/dirs that are inside the directory specified by the enviroment variable?

Further reference: containers/podman#14817 (comment) and containers/podman#14817 (comment)

[mtrmac]

Thanks for reaching out.

Generally, c/image, as a Go library, prefers not to consume environment variables (though there are various quite notable exceptions) because environment variables are invisible to Go callers, and their programs might break without the authors of the programs being aware of anything they could do otherwise. That’s especially the case if new environment variables are added to affect existing functionality.

We do, however, recognize that environment variables are useful for CLI tools, and that it is also useful to be consistent about their use. So, recently we have started adding https://github.com/containers/image/tree/main/pkg/cli/environment, which allows Go callers to opt into using commonly-defined environment variables.

So, for the proposed purpose, the steps would be:

• If the goal is exclusive parsing of specific configuration (which does make a lot of sense, compare Speculative: Make it easier to run Skopeo as a container skopeo#849 ), make sure the individual c/image features can be configured by Go code (probably via types.SystemContext) to use specific files exclusively. I think that’s mostly the case but I didn’t now take the time to review all of the features.
• Then add a pkg/cli/environment function to opt into the new variable.
• Somehow… coordinate that with with the other projects. A naturally-named place for the variable would be c/common, but the dependency chain goes c/storage → c/image → c/common, so it would probably need to start in c/storage. I’m very unclear about this part.

[0x0D15]

Thanks for the detailed explanation, and for your last point perhaps some type of internal coordination at Red Hat would be best here?

[dezza]

Answering related to containers/podman#14817 (comment) (closed)

containers/image (this) seems to be fully XDG_CONFIG_HOME compliant.
https://github.com/containers/image/blob/898f38590573f2801232a74078f2438209d7449a/pkg/docker/config/config.go#L155-L159

containers/storage also seems to take into account.
https://github.com/containers/storage/blob/bf9084b26e028333ee7da21237ae3562e32dfa0e/types/utils.go#L41-L42

Potential patch in podman (handle and support XDG_CONFIG_HOME, would solve need for CONTAINERS_REGISTRIES_CONF):
https://github.com/containers/podman/blob/9f7af81cf8a77f911c9843b1ed88ab8167874ace/cmd/podman/common/registries.go#L13-L22

@0x0D15 @rhatdan
Is it correct assumption that for podman at least- the missing piece is the CONTAINERS_REGISTRIES_CONF logic for supporting XDG_CONFIG_HOME fully ?

@0x0D15 does XDG_CONFIG_HOME solve what you want- if podman became fully compliant with the potential patch mentioned above? I think its a shame if you have to set CONTAINERS_REGISTRIES_CONF when you expect it to load $XDG_CONFIG_HOME/registries.conf as well as $XDG_CONFIG_HOME/registries.conf.d/*

[mtrmac]

I think a lot of relevant experience has been gathered over time, and @Luap99 is now looking at root-only vs. rootless-only vs. root-for-rootless configurations. That might be relevant here — though I’m not sure whether there is a simple consistent story that can be told.

I suspect that, without a concerted effort to change everything, the default outcome will remain that every config mechanism has its own potentially diverging rules.

(@dezza note that overriding XDG_CONFIG_HOME and causing container configuration to be read exclusively from the proposed CONTAINERS_CONFIG_DIR have ~nothing in common. The former does not affect default-consumed paths in /usr or /etc at all.)

[Luap99]

I think my work is largely unrelated containers/common#2499, it simply added new /etc/ lookup locations.

One ask I have seen in the past is to overwrite the config root / dir basically with an env to allow other packager managers use non standard directories for their installs. So then the lookup would be /mycustom/dir/usr/share/containers. I am not to sure how I feel about a env var for such a thing but I think that would cover this ask here except you would need to nest some more directories.

Long term we are looking to rework config files parsing as part of 6.0 so it should be considered for that. I think the biggest isues for our tools right now is that many our config files behaves slightly different, reading order, drop in, XDG_CONFIG_HOME support, etc... So that for me is the main focus to fix. Part of that should be to consolidate the config parsing code. And once we have like one location to define all the paths then it becomes easy to add such an env if we want to support that.

[mtrmac]

So then the lookup would be /mycustom/dir/usr/share/containers

BTW this exists in Go as RootForImplicitAbsolutePaths, but the goal is within-process configuration isolation (e.g. a single large process has two unrelated image use cases, or it wants to set up a configuration itself and explicitly exclude whatever settings root put into /usr).

I don’t think it makes any sense for package installation to ask every single application in existence to consistently add a feature to rewrite its hard-coded path references — after all, that’s what user namespaces and containers are for.

[Luap99]

BTW this exists in Go as RootForImplicitAbsolutePaths, but the goal is within-process configuration isolation (e.g. a single large process has two unrelated image use cases, or it wants to set up a configuration itself and explicitly exclude whatever settings root put into /usr).

Well sure in c/image, not for storage.conf, containers.conf. Which again shows that we need to consolidate the code and make the config file behave in the same way.

Btw how does this work? Doesn't this mean you cannot really cache this file since each API call could specify a different path and then you end up re-reading the file every single time? Doesn't sound very nice performance wise until I am missing something. I see pkg/sysregistriesv2 seem to cache every single path which seems like a memory leak on long running processes that keep using different paths while other config files don't seem to do caching?

I think having one process using multiple different config locations like that seems quite problematic, the sys context should already have most options from the config file as overwrites available? So is this really needed for within process isolation?

I don’t think it makes any sense for package installation to ask every single application in existence to consistently add a feature to rewrite its hard-coded path references — after all, that’s what user namespaces and containers are for.

Well you cannot really nest podman, well you can but then you get to live with the downsides of doing that. And it isn't an options on macOS/windows either. I think the common solution is to have build options.

[mtrmac]

BTW this exists in Go as RootForImplicitAbsolutePaths …

Well sure in c/image, not for storage.conf, containers.conf. Which again shows that we need to consolidate the code

Agreed. I don’t know what a solution would look like, but it is a clearly attractive idea.

Btw how does this work? Doesn't this mean you cannot really cache this file since each API call could specify a different path and then you end up re-reading the file every single time? Doesn't sound very nice performance wise until I am missing something.

Many of the config files are, indeed, read every time. A typical image operation deals with fairly large images, so lots of disk I/O, or network traffic, so I think reading a few small files per operation is very unlikely to be material. (But, to be fair, I think c/image has very little effort spent profiling and optimizing, over time.)

I see pkg/sysregistriesv2 seem to cache every single path which seems like a memory leak on long running processes that keep using different paths

Yes. (IIRC the sysregistriesv2 caching was added for CRI-O, which might have been reading the files on a somewhat hot path.)

A lot of the individual config files were developed ad hoc, under a bit of time pressure, perhaps as third-party contributions … it’s less consistent than one would want. In my wildest dreams, configuration location / parsing / storage / overriding / mandatory options should be an OS-level feature — but efforts in that direction usually fizzle out.

I think having one process using multiple different config locations like that seems quite problematic, the sys context should already have most options from the config file as overwrites available? So is this really needed for within process isolation?

SystemContext only works if you overwrite all fields that could be set from a config file

• If we add a new feature, and a new field to a config file and to SystemContext, an existing caller might break on updating c/image because the feature would be unexpectedly activated and the caller wasn’t yet updated to disable it.
• We might not add every single config file field to SystemContext
• With Go’s weak language features, and us being sloppy, frequently "" is treated as “field not set”, not “field set to explicitly do nothing”. SystemContext frequently does not provide a way to say “don’t read the default config file” only “read a file at some other location”
• (~unrelated: SystemContext, collecting options for all things one might ever want to do in c/image, is a relic from the original CLI history of the codebase; it is not modular, does not scale, can not report ignored options, and over time we should not be moving away from it towards functional options or something like that.)

I don’t think it makes any sense for package installation to ask every single application in existence to consistently add a feature to rewrite its hard-coded path references — after all, that’s what user namespaces and containers are for.

Well you cannot really nest podman, well you can but then you get to live with the downsides of doing that. And it isn't an options on macOS/windows either. I think the common solution is to have build options.

My mistake — at build time, sure, overriding the GNU $prefix makes sense (and a package-specific packaging expert can set that up). I was thinking that a pre-built package should be relocatable at install time, or even at runtime, via an environment variable or something.

[dezza]

@mtrmac Wrong place to fix, I figured it out.

This might have gone unnoticed because the functionality hides itself in what seems like a test-related function from the comment:

containers/image@0ee22b4#diff-522aafc375c87990da7db36f2d4ec60da3b1120aad6b3d18ab0bcf6f1e1bf5caL566-L567

But ~/.config is indeed hardcoded:
https://github.com/containers/image/blob/main/pkg/sysregistriesv2/system_registries_v2.go#L77-L80

Despite being documented here as intended behaviour:

https://github.com/containers/podman/blob/a73aa5710b1a09f05a69fcd79123d6cfe4e97ae3/docs/tutorials/rootless_tutorial.md?plain=1#L165

[mtrmac]

But ~/.config is indeed hardcoded: …

Despite being documented here as intended behaviour:
https://github.com/containers/podman/blob/a73aa5710b1a09f05a69fcd79123d6cfe4e97ae3/docs/tutorials/rootless_tutorial.md?plain=1#L165

The authoritative man page says ~/.config; at least as of the current implementation, that tutorial is wrong, and it should be fixed — unless we change the behavior, of course.