Feature Request: make SitemapGenerator::MAX_SITEMAP_LINKS a configuration...?

[PikachuEXE]

I understand the documented number of links limit per sitemap file is 50,000.
But a SEO team we hired recommends 5k:

You'll want to look into some kind of server-side plugin that can generate sitemap indexes since XML sitemaps should have less than 5k URLs each, but ideally 1-2k

I am not sure if there are enough people needing this, hence putting the question mark.

[PikachuEXE]

I currently just assign to the already defined constant SitemapGenerator::MAX_SITEMAP_LINKS

[kjvarga]

That's the right way to do it until this becomes an option. Would be useful I agree.

[PikachuEXE]

Need a PR?

[kjvarga]

Yes please

On Wednesday, May 13, 2015, PikachuEXE notifications@github.com wrote:

Need a PR?

—
Reply to this email directly or view it on GitHub
#188 (comment)
.

Karl Varga
e: kjvarga@gmail.com
w: kjvarga.blogspot.com
c: +1.250.884.9895
Skype: kjvarga

[PikachuEXE]

Where to put it?

• SitemapGenerator::Sitemap
• SitemapGenerator
• SitemapGenerator.configuration
• SitemapGenerator::Sitemap.configuration
  How to name the config?
• max_sitemap_files

[kjvarga]

So how I want to do this is to add :max_sitemap_files as an option to SitemapGenerator::LinkSet class (including attribute accessors). When that initializes the SitemapGenerator::Builder::SitemapIndexFile in the sitemap_index function, pass the option in to the initializer. Then have the index file use the option and not the global constant.

Groups share the index file, so this way, with multiple groups in a sitemap, the number of files should be constrained across all groups. I'd want to sanity check that things work correctly when SitemapGenerator::LinkSet :create_index option is false. AFAI can remember we create an index file regardless, but just don't write it out, so things should still work, but I'd want to confirm that.

[PikachuEXE]

I am sure I was talking about max_sitemap_links only, not max_sitemap_files
So #198 is opened for max_sitemap_links

[kjvarga]

Ah I got confused by the title of this issue.

[PikachuEXE]

Oops let me update the title

[chrisrichard]

Any chance lowering the max_sitemap_links setting will limit memory use when generating sitemaps with a lot of links? Does the generator write the sitemap files to disk as soon as the limit is reached or is everything kept in memory and written out at the end?

[kjvarga]

@chrisrichard it could help reduce memory use, but I suspect your problem is elsewhere, like how you are loading your ActiveRecord records for example. Each sitemap file is held in memory until it is written to disk, then the memory is freed. The index file has to stay in memory until all sitemaps have been written. If you have lots of AR records, you probably want to use a batched find, cause you might be loading all your records into memory before you start writing the sitemaps.

[kjvarga]

This is almost ready to release #262. I ended up doing a heck of a lot of cleanup in the specs and some other housecleaning so I'm being cautious and testing. Will release it very soon.

[kjvarga]

Released in v5.3.0