Warc writer chain

[nlevitt]

Idea here is to make warc writing configurable such that a custom processor can write a special warc record. The proximate need is to write special youtube-dl json records. Really didn't want to hack something specific to that case into WARCWriterProcessor.

The default chain (see profile-crawler-beans.cxml):

  <property name="chain">
   <list>
    <bean class="org.archive.modules.warc.DnsResponseRecordBuilder"/>
    <bean class="org.archive.modules.warc.HttpResponseRecordBuilder"/>
    <bean class="org.archive.modules.warc.WhoisResponseRecordBuilder"/>
    <bean class="org.archive.modules.warc.FtpControlConversationRecordBuilder"/>
    <bean class="org.archive.modules.warc.FtpResponseRecordBuilder"/>
    <bean class="org.archive.modules.warc.RevisitRecordBuilder"/>
    <bean class="org.archive.modules.warc.HttpRequestRecordBuilder"/>
    <bean class="org.archive.modules.warc.MetadataRecordBuilder"/>
   </list>
  </property>

As an example, to let ExtractorYoutubeDL write youtube-dl json warc records, add this to the chain:

        <ref bean="extractorYoutubeDL"/>

(You will have an identical element in the fetch chain.)

Informative code snippet:

    protected void writeRecords(CrawlURI curi, WARCWriter writer) throws IOException {
        URI concurrentTo = null;
        for (WARCRecordBuilder recordBuilder: getChain()) {
            if (recordBuilder.shouldBuildRecord(curi)) {
                WARCRecordInfo record = recordBuilder.buildRecord(curi, concurrentTo);
                if (record != null) {
                    writer.writeRecord(record);
                    if (concurrentTo == null) {
                        concurrentTo = record.getRecordId();
                    }
                }
            }
        }
    }

We've had this running successfully in QA for 4 months, to the point I had completely forgotten about it, so I think it's pretty much complete. (I had aspirations to refactor some of the messy stuff like managing the dedup metadata in updateMetadataAfterWrite(). But I don't have a specific approach in mind and there's no urgent need.) This code may inherit bugs from legacy WARCWriterProcessor. Probably just needs javadocs 😬

[nlevitt]

Would like to get your eyes on this @anjackson and/or @ato, at your convenience :)

[ato]

Seems like a sensible idea.

The one possible problem that jumps out to me is that some of the record builders make assumptions about concurrentTo being present or not present. It looks fine with the default config but changing the order, disabling a builder or adding a custom one could either cause a NullPointerException or missing header. I think we should probably do the concurrentTo if-statement consistently in every builder.

[nlevitt]

I think this is ready for merge now. Only lightly documented, but maybe enough.