SPIKE: How to handle huge numbers of files?

[bess]

Problem

When we have a huge number of files (>10,000 for working purposes, but we don't know exactly) we cannot display them because the UI can't handle it. We need to decide on a strategy. This ticket is to investigate options and write a decision document with pros and cons of each to take to @matthewjchandler and other stakeholders for feedback.

Notes from the test that spawned this ticket: https://docs.google.com/document/d/19LE4rBmseCckFLga_0hpqZKYvbs4HBJdl0h5ezLTsN0/edit

Proposed solutions:

• Use tar to group the files (Chun will pursue this now)
• Show directory level instead of file level in PDC describe
• Allow the user to download a csv with the file names

Acceptance Criteria

• Create a decision document with a section for each proposed solution
• Check in with Chun about how his investigation into tarring large numbers of files turned out
• Estimate a level of effort for each proposed solution
• Lead a discussion with the RDSS team (plus @matthewjchandler ?) about the pros and cons of each option
• Once a decision is reached, write an ADR outlining the decision
• Make a ticket to implement the decision