Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

configd / custom template overwrites / overlay #1934

Closed
AdSchellevis opened this issue Nov 17, 2017 · 11 comments
Closed

configd / custom template overwrites / overlay #1934

AdSchellevis opened this issue Nov 17, 2017 · 11 comments
Assignees
@AdSchellevis
Labels
feature Adding new functionality

Comments

@AdSchellevis
Copy link
Member

From time to time people ask for functionality that doesn't really fit our core system, but uses parts of the system already in place (which makes it hard/impossible to use our plugin system to extend).

A nice example of such a feature is #1882 , which most likely will not make it into our system because there's no use case for it in scenarios we support.
Other examples are complex suricata scenario's which can't be configured from the gui.

One way of dealing with this is to support overwrites on our templating system to use your own modifications, although there are downsides in doing this it could offer vendors the possibility to maintain their own version of our plugins and users the option to overwrite our defaults.

The addition I would like to propose here is an extension to the +TARGETS file, named +TARGETS.overlay, which can register new targets or overwrite the ones which are already in the normal targets. So, an example:

For squid we would like to create our custom configuration, which overwrites our defaults which are statically configured.
src/opnsense/service/templates/OPNsense/Proxy/+TARGETS.overlay
squid.conf.custom:/usr/local/etc/squid/squid.conf

And the original installed by default.
src/opnsense/service/templates/OPNsense/Proxy/+TARGETS

rc.conf.d:/etc/rc.conf.d/squid/squid
squid.conf:/usr/local/etc/squid/squid.conf
newsyslog.conf:/etc/newsyslog.conf.d/squid

The downsides are:

  • You can only have one overwrite per module, so two packages trying to install an overwrite will fail on install (we won't accept plugins using overwrites, for obvious reasons)
  • Changes on default templates won't propagate to the custom templates, so vendors do need to maintain their own templates.

I will add a commit referencing this issue later on, up for discussion.
@AdSchellevis AdSchellevis added the feature Adding new functionality label Nov 17, 2017
@AdSchellevis AdSchellevis self-assigned this Nov 17, 2017
AdSchellevis added a commit that referenced this issue Nov 17, 2017
@AdSchellevis
configd / template overwrites, for #1934
42f3e75
@fichtner
Copy link
Member

fichtner commented Nov 17, 2017 via email

@AdSchellevis
Copy link
Member Author

I don't mind adding a glob, as long as we suffix them with something that makes it obvious that it is an overwrite (to prevent accidental files from being processed, such as +TARGET.old).

It might increase the possibilities a bit, but there can only be one template per described target (it can't/won't merge)

@fichtner
Copy link
Member

fichtner commented Nov 18, 2017 •
edited
Loading

It doesn‘t have to merge, we just need to make sure our normal templates in +TARGETS are first, and then we can iterate through any number of overrides. But making a safe glob pattern is hard. What if we create a +TARGETS.D directory instead? But what to ignore and what not is still a problem unless we say no file in there is ignored?

@AdSchellevis
Copy link
Member Author

a +TARGETS.D directory sounds good to me, what if we scan for files there that end on .TARGET?

so one can create an overwrite for squid.conf like this:

src/opnsense/service/templates/OPNsense/Proxy/+TARGETS.D\squid.conf.TARGET

@fichtner
Copy link
Member

Lots of semantics for newcomers, but sounds good to me as long as prefix of the file is only cosmetic, not functional (content stays the same)

@AdSchellevis
Copy link
Member Author

Well, the upside is this should only be used by people who know what their doing ;)
I will add a probe for +TARGETS.D directory, contents won't change, with this we have something to point integrators to (without having to discuss pulls that won't make it).

Thanks for your feedback

AdSchellevis added a commit that referenced this issue Nov 19, 2017
@AdSchellevis
configd / template overwrites, for #1934
735f9b8 
as discussed with @fichtner use +TARGETS.D to host the target overwrite tags
@AdSchellevis AdSchellevis mentioned this issue Nov 19, 2017
Closed
fichtner pushed a commit that referenced this issue Nov 22, 2017
@AdSchellevis @fichtner
configd / template overwrites
6ade0c6 
PR: #1934

(cherry picked from commit 42f3e75)
(cherry picked from commit 735f9b8)
@chelming
Copy link

chelming commented Dec 28, 2021 •
edited
Loading

so one can create an overwrite for squid.conf like this:

src/opnsense/service/templates/OPNsense/Proxy/+TARGETS.D\squid.conf.TARGET

is this working as intended? it appears that squid.conf.TARGET will be read as a target and look for paths specified in it instead of being read as a template to override squid.conf located a directory above?

the actual practice appears to be:
Inside of +TARGETS.D create a file like override.TARGET containing

squid.conf.custom:/usr/local/etc/squid/squid.conf

and creating squid.conf.custom in the src/opnsense/service/templates/OPNsense/Proxy directory.

Based on the conversation in here and on the target overwrites page I was assuming that any matching file + '.TARGET' in the +TARGETS.D directory would override the file one directory higher but that appears to not be the case. I'm not sure what the benefits of the +TARGETS.D directory would be otherwise vs just using a file like +TARGETS.overrides that contained target overrides.

@AdSchellevis
Copy link
Member Author

@chelming

the actual practice appears to be:
Inside of +TARGETS.D create a file like override.TARGET containing
...

That is indeed the intended purpose, overwrite definitions with custom ones.

core/src/opnsense/service/modules/template.py

Lines 90 to 93 in 88ab77f

target_sources = ['%s/+TARGETS' % file_path]
if os.path.exists('%s/+TARGETS.D' % file_path):
for filename in sorted(glob.glob('%s/+TARGETS.D/*.TARGET' % file_path)):
target_sources.append(filename)

... I'm not sure what the benefits of the +TARGETS.D directory would be otherwise vs just using a file like +TARGETS.overrides that contained target overrides.

Overwrite system defined or plugin defined configuration files with those of your own. As mentioned in the docs, handle with care as the original configuration file is likely subjected to change. If one would like to ship a custom squid.conf for example, you can divert the configuration output with one of your own.

@chelming
Copy link

chelming commented Dec 28, 2021 via email

@AdSchellevis
Copy link
Member Author

Right, I get the purpose, but what's the difference between having separate
+TARGETS.D/users.conf.TARGET and +TARGETS.D/base.conf.TARGET files vs one
single +TARGETS.D/overrides.TARGET file with definitions for base.conf and
users.conf like the +TARGETS file?

There isn't, all files are processed in order so different "vendors" can plug different files from the base target definition file.

@chelming
Copy link

chelming commented Dec 28, 2021 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@AdSchellevis AdSchellevis

Labels
feature Adding new functionality
Milestone
No milestone
Development

No branches or pull requests
3 participants
@fichtner @chelming @AdSchellevis