Support for robotsflag field based on OpenWayback behaviour

[anjackson]

We were looking at using the <robotstxt> field, which is currently not supported by tinycdxserver, and were wondering if you'd be happy for us to submit a pull request that enables it?

The implementation in OpenWayback is rather odd, in that it populate this field using the M meta tags (AIF) field (see here). It's not clear why the meta tags field becomes the robotstxt field, but AFAICT this is the only way to populate that field via the CDX format.

It doesn't look like too difficult a change, but given that it's nearly there but commented out I thought I'd better ask if there's a problem? Presumably the indexes won't be compatible either?

[ato]

Sure, would be happy to accept a pull request that implements it.

There's no problem with it. The index format includes a pre-record (per CDX line) version number. So create a new ddcodd

[ato]

Gah. Sorry. 'close issue' is too near the text box on mobile.

... So create a new Capture.decodeValueV2() method for a version 2 record format that supports the robots field and update Capture.encodeValue() to write the new format. Then the index server will happily read both new and old records and you can even mix them in the one index while incrementally reindexing to fill in the robots field data.

It was marked as todo simply because I didn't have any CDX files on hand with that field populated and wasn't sure what the data format was or what exactly it was used for in Wayback.