DURACLOUD-1268: refactor the RetrievalTool to create full list of contentIds, including chunked content, when a list-file is used

[nwoodward]

This PR changes the DuraStoreSpecifiedRetrievalSource class so that when a list-file is specified it creates a full list of contentIds to be retrieved, including any chunked content of files in the list-file.

---

JIRA Ticket: https://duracloud.atlassian.net/browse/DURACLOUD-1268

What does this Pull Request do?

This PR changes the class used by the RetrievalTool when a list-file is specified to work more like the default behavior when retrieving a space. As as result of this change, the two classes can now retrieve content with the same function. This PR resolves the case where the RetrievalTool with a list-file specified fails to transfer when restarting from partially chunked files.

How should this be tested?

• Create a space in DuraCloud and use the SyncTool to add one or more large files that will be chunked
• Use the version 7.0 of the RetrievalTool with a file containing the contentIds in the space to start retrieving the space
• Hit Ctrl-C to stop the process. Restart the process and observe that it fails to retrieve the files
• Repeat the above process with the RetrievalTool built from this branch, and observe that the process finishes successfully

Interested parties

@duracloud/committers

[bbranan]

Thanks for putting this together Nick! This approach seems like a good one. I've made a few suggestions. My primary concern is that we don't end up dropping files from the retrieval list if they haven't yet made it into the space manifest (more details in the specific comment).

It's worth noting that this change will result in no longer being able to request a list of chunk manifest files (or chunk files) to be downloaded independently by the Retrieval Tool. While this need doesn't come up often, I know that this approach has been used to retrieve all chunk manifests in order to perform a final verification of chunk transfers. I still think this is the right direction, I'm just calling out a change that may otherwise be unexpected.