Should we have a better mechanism for setting RDF::URI.cache capacity?

[no-reply]

It is currently possible to override RDF::URI::CACHE_SIZE directly with = (with warnings printed to STDOUT) only before the cache has been initialized. There's not a clean interface for setting up the cache when it has already been touched (e.g. in a Rails initializer).

Should we be thinking about better ways to control the cache size during application startup or at runtime?

[gkellogg]

+1 Providing a clean mechanism to override this when an application starts up makes sense.

[no-reply]

Assigning to self. I'm not setting a 2.0 milestone, since I think this could be safely introduced in 2.1.

Still, I may try to get it in for 2.0.

[floze]

Hey there, chiming in with some user feedback.

When processing large amounts of scraped website data, the inability to set a limit to the cache piles up a huge amount of memory over time. In fact, the only way to mitigate is to periodically restart the underlying process, or to monkey patch and force-override the cache capacity. Overriding RDF::URI::CACHE_SIZE alone as a workaround did not work for us, as e.g. rdf-rdfa (and potentially anything that utilizes RDF::Util::Cache) initializes the cache without limits regardless.

So from my point of view, to save some RAM, this feature is very welcome! There should be an option to "hard-limit" the cache capacity.

[gkellogg]

@floze any suggestions on how to do this cleanly? It would certainly be a useful feature. The RDF::Util::Cache API allows you to specify capacity, but it't typically used from RDF::URI using the CACHE_SIZE constant.

We could consider an environment variable, that would override this, but something would be necessary in every place which uses it. Alternatively, the CACHE_SIZE constant, itself, could be set from an environment variable and default to -1, which would have universal effect.

[floze]

Other APIs I frequently use, like the ruby AWS SDK for example, go both ways and allow for an environment variable to be set (e.g. AWS_REGION), and to configure the API dynamically using a shared config as well. For example you may call:

Aws.config.update(region: 'us-west-2')

anywhere and you're good to go. That's pretty convenient if you're just fiddeling around, or using initializer scripts to configure your application (although it may get hairy if you want to differentiate several environments, like development and production). The environment variable approach otoh is very convenient if you're using, for example, a docker setup, where you simply cram in a bunch of env vars for each container.

So from my point of view I would love to have both actually. Setting RDF_CACHE_SIZE as an env var and calling RDF.config.update(cache_size: 123) would both be terrific.

However, how would you go about making sure the capacity limit is always being respected without altering RDF::Util::Cache's API? Having an env var or user config take precedence over instantiation/initialization (see link in my post above) would be rather awkward, wouldn't it?