Fix resetting metadata on many repositories at once via the shed API

[jmchilton]

The web controller method worked but the API method would collapse the revisions with metadata down to a single one. I suspected this when I opened #14890 but now I've brought in some test infrastructure from a downstream working branch to verify the bug and verify the fix. I've placed the tests and the fix in separate commits so the broken can be easily verified and so that we can backport the fix without tests if we'd like.

More About the Testing Code

One of my medium terms projects is to eliminate the complexity in the tool shed that allows uploading individual files and such to a repository. Planemo only uploads full archives and I would like the API for the tool shed to be this. This is how... all package managers work and since the introduction of Planemo we've been working to get away from the tool shed taking on other roles (i.e. source control management). This should simplify the tool shed and simplify what an alternative backend would have to support. In that direction, I would like to be able to quickly define like test repository lineages that can be uploaded for testing (both functional and unit tests). The old tool shed test code had some efforts in this direction but not as systematic and using tar files (which are difficult to work with and not ideal for sticking in git). All of that is to say that the column maker tools here are not just for these test cases but are about migrating the existing tests in better more manageable directions.

More About the Bug

rmm.set_repository resets resetting_all_metadata_on_repository and without that set the bug is expressed. This is troubling state management from a design perspective but I'm not going to fix it in a bug fix and is sort of endemic to these very stateful tool shed utilities (on the client and server side). The bug isn't exhibited for the web controller version of this functionality used by the UI because:

                    repository = repository_util.get_repository_in_tool_shed(self.app, repository_id)
                    self.set_repository(repository)
                    self.resetting_all_metadata_on_repository = True
                    self.reset_all_metadata_on_repository_in_tool_shed()

The consumer of the functionality re-resets the value back to the original after setting the repository.

How to test the changes?

(Select all options that apply)

• I've included appropriate automated tests.

./run_tests.sh --toolshed lib/tool_shed/test/functional/test_shed_repositories.py:TestShedRepositoriesApi.test_reset_all

License

• I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

[mvdbeek]

I've looked at it couple time now, and I haven't figure out why exactly this fixes the bug. Is it because there's still something that set_repository didn't correctly reset to the right repository ? Given all the things the init method does this is definitely better, just curious at this point.

[jmchilton]

😆 It isn't obvious at all - it took hours of debugging. set_repository resets resetting_all_metadata_on_repository to False, which has... implications for how working directories are used in generate_metadata_for_changeset_revision (in metadata_generator.py). When it resets, the actual directory of the repository is used and only the most recent contents of the repository are used as the working directory to take metadata from. So the tool shed is like "heres is all my tools and versions" over and over for each revision but the contents used to generate that are only the latest revision... and so only one interesting metadata repository is discovered.

[mvdbeek]

Unsolicited advice: I often log in and sniff around by hand, https://github.com/mxschmitt/action-tmate#use-registered-public-ssh-keys is super useful for that.