Bagit export updates

[rstorey]

Closes #664

[rstorey]

In the interest of making this project more reusable, should we have this as something like a dictionary in the settings file?

It seems that our BagIt export (at least for now) is going to be strictly for the purpose of getting transcription data into www.loc.gov. The most recent changes make it even more LC-specific.

When we are ready to talk about/implement public exports, I'm not sure that BagIt 1.0 bags are what we want to use. I think the CSV export is better, not the least of which is because it's a streaming response that won't OOM on large exports.

[coveralls]

Coverage decreased (-0.4%) to 71.414% when pulling 292a451 on bagit-export-updates into 7121c49 on master.

[acdha]

Do we need to apply published checks here to exclude the campaign/project/item/asset(s) which aren't published yet?

[acdha]

This will fail if it gets a value which doesn't match the pattern. We should probably do the if not match: … check even if that's unlikely.

[rstorey]

I changed it - let me know what you think.

[acdha]

When we are ready to talk about/implement public exports, I'm not sure that BagIt 1.0 bags are what we want to use. I think the CSV export is better, not the least of which is because it's a streaming response that won't OOM on large exports.

I suspect that the best answer here is going to be Celery and the work on bagit-python to make it upload directly to S3 (and that could be safely made public with some sort of throttle). That's definitely a separate project but we might want to see whether there's something we can do here to avoid OOMs — do you know where in the code it's hitting that limit?

[rstorey]

When we are ready to talk about/implement public exports, I'm not sure that BagIt 1.0 bags are what we want to use. I think the CSV export is better, not the least of which is because it's a streaming response that won't OOM on large exports.

I suspect that the best answer here is going to be Celery and the work on bagit-python to make it upload directly to S3 (and that could be safely made public with some sort of throttle). That's definitely a separate project but we might want to see whether there's something we can do here to avoid OOMs — do you know where in the code it's hitting that limit?

I haven't actually personally witnessed an OOM, so not sure.

[rstorey]

SInce we're only getting completed transcriptions in the BagIt export, either the assets are published or were published. I think the scenario where something is published, completed, unpublished, and then exported might be a valid use case, so I'm not going to add a filter for publish status at this time. If @elainekamlley wants to weigh in, I defer to them.

[acdha]

This could be some other exception class but I care less about that than that it’s checked since it seems like this condition will usually be telling us that we got some bogus data from the API, and that edge case probably means we need to get some records fixed

[acdha]

I’m cutting my review short for family reasons but it looks good enough to me to merge for testing