Watcher is not detecting updated files when SE is fixing issues

[benoit74]

On April 2, SE pushed the March 2024 dumps to archive.org ; the watcher retrieved them and pushed them to S3.

SE quickly realized that the dumps were "corrupted/buggy".

They pushed an update on April 7.

See https://meta.stackexchange.com/a/398606 for details.

The watcher never retrieved these updated files and this caused issues on our side:

• the scraper was not working anymore
• I failed to realize this was just a corrupted dump issue and already fixed upstream

This already happened in the past.

The watcher should probably be smarter at detecting changes, so that we have higher chances to process the last available dump. The last available dump might still be corrupted, but at least it is the "best dump so far", and I won't say "this dump is 1 month old, it is obviously used by many users, it should be ok or issues would have been reported upstream".

WDYT?

[benoit74]

Btw, this could be an elegant solution to fix the fact that all dumps currently stored on S3 are still from April 2 and "corrupted"

[rgaudin]

So it's clear for everybody, the watcher didn't pick up the fixed version because we identify a release as a YYYY-MM string given updates are expected every quarter.

This string is computed from the Last-Modified header of the Sites.xml file and compared to the one we stored in S3.
An update within the same month cannot be detected this way.

[benoit74]

Here the dumps have been updated but the Sites.xml has not been updated at-all, so current watcher behavior is not aligned (anymore) with SE practices.

I suggest to:

• stop checking the version of "Sites.xml" (we do not care about it indeed)
• check the Last-Modified header of every individual archive
• store the whole Last-Modified header in S3 metadata (instead of the simplified version)
• use this whole Last-Modified header for taking the decision to download the archive again