Where should we collect example files?

[marhop]

Hi,

while working on documenting error messages in the By JHOVE! Explain Yourself group I wondered whether we need some guidelines on what to do with example files that exhibit a specific error. Uploading or linking them to the respective wiki entry is easy enough, but should they also somehow be integrated into JHOVE's test corpus? I guess that's a little more involved though ...

Cheers,
Martin

[asciim0]

Great questions! Within the format-corpus repo, there's also a (currently empty) folder called "jhove-errors". maybe we can make use of that and create subdirectories for the modules and then the errors?

[asciim0]

@carlwilson and @darrendignam : can we add sample files to e.g.
https://github.com/openpreserve/jhove/tree/integration/test-root/corpora/errors/modules
or
https://github.com/openpreserve/format-corpus/tree/master/jhove-errors
?
We would suggest a structured approach using module directories and error-id directories, e.g. /TIFF-hul/TIFF-hul-6 and then link the sample files from the jhove github wiki as well.

[carlwilson]

Hi @marhop at first I would start by uploading to the Wiki. The corpus for testing has some structure but I may tweak it before opening to all comers. The Wiki is a nice safe starting space. Many thanks for this.

[marhop]

The JHOVE wiki at https://github.com/openpreserve/jhove/wiki

IIRC it's possible to not only link but also upload files there that will then be stored in some amorphous GitHub storage. That's OK to get started although in the long run I think something with more structure like the JHOVE test corpus mentioned above will provide a better overview.

[asciim0]

I want to circle back to this discussion.
@marhop have you actually uploaded any test files to the wiki?
IMO it's not quite ideal, as you'd have to look through all the pages to see where there are sample files. We also need to take into consideration, that error messages for files sometimes change with fixes or files have multiple error messages and therefore i don't find an upload to the wiki ideal.
Curious to hear your thoughts on this.

[marhop]

Ha, no, I have not, because it's not possible!

While all your points about uploading files to the wiki being very unstructured and difficult to navigate are perfectly valid in themselves, there is yet another thing that speaks against this approach: it's not possible.

When I said above that "IIRC" it's possible to upload files to the wiki I did in fact not remember correctly. It's possible to upload files to issues (which I have frequently done) but as it seems, the same is not true for wiki pages - at least not in general. As it seems, you can upload some image file formats, but that's all. So not really what we are looking for here. In general, you can only link to arbitrary files, but those have to be hosted somewhere else.

This leaves us with the option of storing those files in JHOVE's format corpus as mentioned above, right?

[asciim0]

yeah, that sounds like a showstopper :-P

and indeed, my vote is still for https://github.com/openpreserve/format-corpus/tree/master/jhove-errors

[marhop]

Sounds good!

[marhop]

OK, so I made a bunch of updates to the TIFF-HUL wiki page in the last week, including links to example files. In lack of a better place I put these files into a repository of my own that is only remotely related to JHOVE and linked to this repo from the JHOVE wiki (example wiki entry). (Never mind that the example files are Markdown, not TIFF - this doesn't make a difference for the question at hand.)

While this approach works it is very far from perfect. What if I decide to delete this repository and forget about the files therein being linked from the JHOVE wiki? The same problem obviously exists with any externally linked files. So I think we should really move towards https://github.com/openpreserve/format-corpus/tree/master/jhove-errors as proposed by @asciim0.

[BertrandCaron]

@asciim0 @carlwilson

I also would like to add some files to that corpus. Should we fork it and make a pull request or be granted push privileges? The URL http://wiki.curatecamp.org/index.php/Collecting_format_ID_test_files in the 'How to contribute?" section is broken!

[asciim0]

please put the sample files here: https://github.com/openpreserve/format-corpus/tree/master/jhove-errors

[asciim0]

We re-discussed this in the JHOVE IG chairs meeting - please put the sample files here: https://github.com/openpreserve/format-corpus/tree/master/jhove-errors

[mutanthumb]

Hi-
I ran JHOVE 1.34 on my pile of 175,014 PDFs. I have attached the results (jhoveXMLout.csv). There were 57 different PDF-HUL messages. I am happy to upload examples for any or all of these PDF-HUL messages. For some messages, there are only 1 or 2 PDF files. How many example files would you like if there are more? Also, would you like new example files for messages if you already have files?

Thanks,
susan

[ross-spencer]

@mutanthumb in case it helps (or maybe others who find this thread later on who might need a hand with GitHub), Carl made a video a while back: https://www.youtube.com/watch?v=TrGTm2qTB3s

[mutanthumb]

Ah, yes, makes sense, thank you!

[mutanthumb]

Pull request has been made. I used GitHub Desktop to process the uploads because the browser kept hanging.

[mutanthumb]

I would be happy to update the PDF-HUL wiki with links to the new examples as well, but I don't have access to do this.

[BertrandCaron]

Excellent!

Now that I looked at the complete instructions for sharing files, I'm wondering if I can re-upload files from the Stressful PDF corpus to the OPF repo, given that they should be assigned a CC0 license... Any thoughts on that?