mcollective plugins create many http requests due to many file resources

[bastelfreak]

Hi!,
this module has a defined resource which is used by all mcollective/choria plugins. The main purpose is to copy a file from the plugin module to the puppet agent. With multiple plugins installed, this leads to a huge number of file resources which gets multiplied with the number of puppet agents. This itself isn't a problem. But if the file resource uses source, each resource creates an additional http call which adds heavy load to the puppetservers.

The current best practice for file resources is to use content => file() which reads a file from the files/ directory of a module and inlines it into the catalog. This has two advantages:

• You can easily write unit tests for the catalog to verify the file content, because the content is now in the catalog
• Heavily reduce the amount of http calls

[ripienaar]

Can't say I am aware thats the current best practise? it would lead to huge catalogs and big increases in memory use in the agent.

[bastelfreak]

it's always a trade off. Some people tend to sync huge binary blobs via puppet. In those cases it's probably better to stick with source (or to simply avoid doing this). It was mentioned multiple times by Puppet employees on Slack/IRC. It still seems to be the case that puppetserver 6.14 doesn't scale properly with the amount of http calls. I tested this locally and couldn't detect an increased load on the agents, but a reduced load on the puppetservers.

[ripienaar]

I was working on shipping plugins as part of the ruby compat gem which would long term be better, but not really worked on that for a while :(

So maybe we make it a configurable behaviour? So people can pick an approach for now

[bastelfreak]

sure, I can implement a param to switch to content instead of source.

[ripienaar]

thank you