Split IO #2

[francisco-dlp]

Description of the change

Split HyperSpy's IO plugins into a separate package RosettaSciIO. See #1978 for the related discussion.

This supersedes #2174

Progress of the PR

• Change implemented (can be split into several points)
  • Move all plugins to a separate folder, with its own setup.py, that must be installed separately.
  • Move each plugin to a folder of its own
  • Move all IO tests to rsciio
  • Enable lazy loading of the plugins by moving the specifications to yamls files.
  • Make rosettasciio fully independent of HyperSpy
    • Don't use DictionaryTreeBrowser. DTB was replaced by python-box
    • Move misc.io code to rsciio
    • Move exceptions to rsciio.
    • Move remaining imports to rsciioMost of it is done, except for:
      • rgb_tools
      • emd relies on hyperspy for reading
    • Make _hierarchical fully independent of HyperSpy
    • Pass dictionary instead of signal to file_writers
      • msa
      • tvips
      • semper
      • blockfile
      • emd
      • panta rhei
      • nexus
      • image ([SPLIT-IO] Make file_writer work with dictionary francisco-dlp/hyperspy#51)
      • mrcz ([SPLIT-IO] Make file_writer work with dictionary francisco-dlp/hyperspy#51)
      • ripple ([SPLIT-IO] Make file_writer work with dictionary francisco-dlp/hyperspy#51)
      • nexus ([SPLIT-IO] Make file_writer work with dictionary francisco-dlp/hyperspy#51)
      • tiff ([SPLIT-IO] Make file_writer work with dictionary francisco-dlp/hyperspy#51)
      • usid ([SPLIT-IO] Make file_writer work with dictionary francisco-dlp/hyperspy#51)
      • zspy
      • hspy
  • Move User Guide IO chapter to rsciio
  • Remove dependency on traits (basically for t.Undefined)
  • Make tests independent of hyperspy (not strictly required for release)
• update docstring (if appropriate),
• update user guide (if appropriate),
• add an changelog entry in the upcoming_changes folder (see upcoming_changes/README.rst) -- see Merge RELEASE_next_minor into RELEASE_next_major #2979,
• Check formatting changelog entry in the readthedocs doc build of this PR (link in github checks)
• add tests,
• ready for review.

How to test it

To test it, simply install rosettasciio (currently at hyperspy's root folder, it'll move to its own repository once there are no more code exchanges between hyperspy and RosettaSciIO) in addition to hyperspy. Everything should work as usual.

Additionally, it is possible to import the readers directly from RosettaSciIO as follows:

from rsciio.msa import api
api.file_reader("your_msa_file.msa")

[ericpre]

Considering that there is not much left to do, I would prefer to finish in the next few days. After that, I am happy to give a go at splitting! :)

[jlaehne]

Considering that there is not much left to do, I would prefer to finish in the next few days. After that, I am happy to give a go at splitting! :)

+1 - would be great if you could do so!

@pietsjoh will be starting to contribute new readers in the next weeks and it would be great to do so directly to the new repo.

P.S.: I am available until Tuesday to review the remaining steps.

[ericpre]

The remaining tasks are done in francisco-dlp#58.

@francisco-dlp, just in case, we can merge and push to your branch, would have time to review and merge francisco-dlp#58? The change are mostly trivial.

Worst-case scenario, we create another branch but it would be better to keep it all in this PR!

[jlaehne]

Just realized that

1. We are missing a changelog entry for this PR.
2. v1.7.1 was not automatically merged from RnP into RnM and RnM is still on 1.7.1.dev0 - including the changelog.

I would propose merging this one nevertheless and sorting out anything related to the v1.7.1 merge into RnM in a separate PR, possibly including the changelog entry of this PR (which should actually be about the split and not about the preparation of the split).

[ericpre]

Yes and there is some tidying up needed on the RELEASE_next_major branch too!

[ericpre]

For the record, here are the step that I did to split into a separate repository:

Command to split the repository

`git filter-repo --path rosettascio/ --path rosettasciio/ --path hyperspy/io_plugins/ --path hyperspy/misc/io/ --path hyperspy/tests/io/ --path doc/user_guide/io.rst --path-rename rosettasciio/: --force

I needed to add all relevant paths to keep the history after folder renames, etc.

Tidy up left over code

Since it was necessary to also filter doc/user_guide/io.rst to keep the history, I remove it in a commit.

Clean some large file from the history

We had some large which were added and removed from code during a pull request and never cleaned up. This was possible to find them using the following command:

git rev-list --objects --all |
  git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' |
  sed -n 's/^blob //p' |
  sort --numeric-sort --key=2 |
  cut -c 1-12,41- |
  $(command -v gnumfmt || echo numfmt) --field=2 --to=iec-i --suffix=B --padding=7 --round=nearest

ref: https://stackoverflow.com/questions/10622179/how-to-find-identify-large-commits-in-git-history

The largest file before history cleaning were:

3c272c431de2  8.7MiB rsciio/tests/emd_files/fei_emd_files.zip
b4dce2fe8c67   10MiB hyperspy/tests/io/sur_data/test_spectral_map_compressed.sur
5fdd25abfa0b   11MiB hyperspy/tests/io/edax_files/spd_map.spd.gz
f89958aa88ac   11MiB hyperspy/tests/io/edax_files.zip
48e1eba14394   21MiB hyperspy/tests/io/nexus_files/file2.nxs
b949accc2fcf   32MiB hyperspy/tests/io/sur_data/test_spectral_map.sur
4a8123661a42   35MiB hyperspy/tests/io/edax_files.zip
502c66aed3c6   35MiB rsciio/tests/edax_files.zip
e5ad43b77116   56MiB hyperspy/tests/io/nexus_files/file1.nxs

And are now

3c272c431de2  8.7MiB rsciio/tests/emd_files/fei_emd_files.zip
5fdd25abfa0b   11MiB hyperspy/tests/io/edax_files/spd_map.spd.gz
f89958aa88ac   11MiB hyperspy/tests/io/edax_files.zip
4a8123661a42   35MiB hyperspy/tests/io/edax_files.zip
502c66aed3c6   35MiB rsciio/tests/edax_files.zip

I keep a local copy of the git repository before in case something was incorrect! After hyperspy 2.0 is released, we should possibly do the same to the hyperspy repository.

[jlaehne]

Thanks Eric for finishing it up!