OOM errors in Docker

[arbythree]

We're seeing fairly heavy RAM consumption when performing basic queries:

@tests = TestProduct.all.order('name').load

We suspect this is due to eager loading, as TestProducts have a number of associations (only a handful of actual TestProducts exist). We've tried suppressing eager loading as described here, but no dice.

Any ideas?

[dlitvakb]

Hey @raybradley,

Looks like you may have multiple circular references. To limit them, either enable reuse_entries or limit the maximum include resolution depth with max_include_resolution_depth, I'd suggest to your maximum include level + 1.

Both of this options are available through the options in the configuration block.

Cheers

[arbythree]

I'm afraid I'm missing something here. Given this config block:

ContentfulRails.configure do |config|
  config.access_token = ENV['CONTENTFUL_ACCESS_TOKEN']
  config.preview_access_token = ENV['CONTENTFUL_PREVIEW_ACCESS_TOKEN']
  config.space = ENV['CONTENTFUL_SPACE_ID']
  config.default_locale = "en-US"
  config.max_include_resolution_depth = 2
end

...the app complains thusly:

undefined method `max_include_resolution_depth=' for #<ContentfulRails::Configuration:0x00007f82c8b59a80>

Ideas? Is the option documented as part of the contentful_rails gem and I'm missing it?

[shaug]

ContentfulRails.configure do |config|
  config.access_token = ENV['CONTENTFUL_ACCESS_TOKEN']
  config.preview_access_token = ENV['CONTENTFUL_PREVIEW_ACCESS_TOKEN']
  config.space = ENV['CONTENTFUL_SPACE_ID']
  config.default_locale = "en-US"
  config.contentful_options = {
    max_include_resolution_depth: 2
  }
end

I personally recommend enabling reuse_entries instead.

[arbythree]

Thanks @shaug ... that was non-obvious :)

Why do you recommend reuse_entries over max_include_resolution_depth?

[dlitvakb]

Hey @raybradley,

It depends, reuse_entries will be more performant and will take less memory, but this is at the cost of disabling caching capabilities when having circular references, as it will generate a stack level too deep error.

Reducing the maximum include resolution limit, while a bit less perfomant when having circular references, has the same performance and uses the same amount of memories when dealing with unique objects. When having circular references, as the referenced objects do not point to the same address and are actually limited in depth, you can safely use caching mechanisms without being afraid of getting stack too deep exceptions.

Hope that explains it a little bit better.

Cheers

[arbythree]

Thanks for the suggestions, but unfortunately it's still happening. Anything further you all would recommend to troubleshoot or profile the calls I'm making?

[dlitvakb]

Hey @raybradley,

I'd suggest using something like NewRelic to check what's the memory bottleneck you're having. So that we can have a better idea of what may be the cause of your issue.

In the meantime, I'd suggest limiting your request size to an amount you can handle with the memory constraints you have. You can do so by using the limit query operator, which defaults to 100 (same if omitted as you currently have). And start playing both with that and with the include operator (if you're using version 1.0 of contentful_model it will default to the nesting level of your models, though this nesting has to be explicitly declared with has_one, has_many, etc relationships - maximum of 10).

Hope this helps,

Cheers

[keithm-thompson]

Hey @dlitvakb,

Looks like adding both reuse_entries and max_include_resolution_depth did the trick for us. Is there any reason why we should avoid having both flags set in the config options?

Thanks!

[dlitvakb]

Hey @keithm-thompson,

They serve different purposes, so you can safely use both together. The first is for avoiding recreating entry objects if they are reused somewhere along the hydration process, the second one is to limit the amount of chaining on long chains of unique objects (if you enable reuse_entries).

Cheers