As a researcher, I want to be able to get the text corpus for a subset of record IDs so that I can conduct textual analysis within particular groups of texts.

[mnaydan]

acceptance criteria

• filter script successfully filters the corpus and generates a subset corpus that includes all pages for the specified source ids and only pages for those source ids
  • for simple idfile (one source_id per line; non-excerpt sources)
  • for simple idfile with leading whitespace
  • for simple idfile with trailing whitespace
  • for idfiles with sources with 1 or more excerpts
  • for idfiles with a mix of existing and non-existing sources
• filter script has reasonable error handling (missing/empty id file, output file already exists)

[rlskoeser]

Here's a screen recording of my terminal showing the corpus subset functionality:

Screen.Recording.2024-04-02.at.2.34.46.PM.mov

[laurejt]

Should an output file be created if there are no matches?

[rlskoeser]

Should an output file be created if there are no matches?

I guess ideally not, but I don't think there's an easy way to detect this until after consuming the generator. I guess we could check and remove it after if it's zero size.

[laurejt]

Testing acceptance complete. The only outstanding issue, is whether the script should produce empty output files when no matches are found (either way seems reasonable).

[rlskoeser]

Thanks for testing. Let's leave the empty file behavior as it is for now, we've both put enough time into this feature already. We can always revisit and tweak it later if we find it's causing problems.