store full original path as archival record

[cccs-ip]

the path is useful as an archival record. otherwise, we would store directory names as attribute tags and perhaps eventually change the folder structure

[pwhipp]

Any document can have an arbitrary number of filenames associated with it in its metadata.

[cccs-ip]

Cool, and thanks. After we run the sha process, can we compile all the different names into all manifestations of the same file?

[pwhipp]

We can easily run a process that deletes the duplicate files and collapses the documents down to one with the other filenames recorded within it. If we do that, the hard bit will be deciding which categorization we keep.
If the categorizations are all valid, these could be added to the document, making the one we choose to keep arbitrary.
For example, if /a/foo.pdf and /b/bar.pdf are identical files, We could end up with one of the following metadata blocks:

1. {filenames: [/a/foo.pdf, /b/bar.pdf], categories: [a, b]}
2. {filenames: [/a/foo.pdf, /b/bar.pdf], categories: [a]}
3. {filenames: [/b/bar.pdf, /a/foo.pdf], categories: [b]}

(1) seems to be the logical choice.

This does not consider the possibility that other metadata could differ (e.g. through a spreadsheet import). If that is the case then both metadata blocks should probably remain (as an optimization, they could be modified to point at the same actual s3 file).

[cccs-ip]

Thanks, Paul. Option 1 and the multiple metadata files where information is conflicting sounds like the way forward.