Zenodo DOI for lumispy

[jordiferrero]

Hi all,

I was recently discussing on setting up a Zenodo DOI for any publication that may come from the code we are actively writing.
I have seen that both hyperspy and pyxem use such a system which keeps track of the versions for referencing.

Before I set up a Zenodo account for this repo, I wanted to discuss with you @jlaehne @ericpre if you have any objections/suggestions before going ahead.

Since we don't have any version released yet (which we should focus on soon, I think), will that be an issue with getting a DOI?

Thank you

[jlaehne]

Yes, we should ... both get a Zenodo DOI and do a 0.1 release.

I have no experience with either (So I do not know if one should predate the other). But am happy to contribute.

[ericpre]

To get a zenodo DOI, it may be need a release first. Maybe it may be worth doing two releases in a row in order to get the concept DOI in the readme and also check that the process are working fine.

Relevant link to what is done for hyperspy:

• https://github.com/hyperspy/hyperspy/blob/RELEASE_next_minor/releasing_guide.md
• https://hyperspy.readthedocs.io/en/latest/citing.html

Your zenodo needs to be connected to your github account and you should be able to enable automatic DOI mining on github release - which is difference from git tag. In the hyperspy release workflow, there is an action which creates the github release when a tag is pushed to the hyperspy/hyperspy repository.

Note that there is a concept DOI and a per release DOI. In the readme, we include the concept DOI, because including the DOI of the each specific release is an egg and chicken issue: the DOI can't be mined before the release is done and the release can't be changed after it has been... released!

There are a number of details, which are easy to miss, it is not very complicated, just tedious and not always completely documented, it not out of date, etc... Happy to help, if you need!

Is it worth setting up an github action to automate the release (create sdist, wheel packages, test these packages and upload to pypi) as we do for hyperspy & co? See, for example: hyperspy/hyperspy_gui_ipywidgets#33

[ericpre]

This link summarises well the process:
https://genr.eu/wp/cite

[jordiferrero]

Thanks for the help!
Two questions:

1. Can I create a concept DOI without the first release? Anyhow, that is the DOI that we would normally reference... Or do we need the first release to connect to Zenodo (as I understand, that's not the case). If not, I would just go ahead now and create the concept DOI.

2. For setting up the first release I am a bit confused with all the wording in https://github.com/hyperspy/hyperspy/blob/RELEASE_next_minor/releasing_guide.md (I know it's not complicated, it's just I never did it before).
   Is it as simple as creating a branch from one of our personal branches? How is the changelog file created? Do we manually need to create a pyPi/conda-forge connection?...

Thank you again!

[ericpre]

1. I don't know, but what would it be referencing then, if there is no release?
2. There is a bit of jargon... the changelog is created manually and is commited as any other code. It needs to be part of the tag/release.

The minimum workflow is as follow:

• have the code ready, including changelog
• make sure the version is set correctly
• make the sdist and wheel - there are different way to do it...
• upload to pypi manually

The two last steps can be done using github actions, which I would recommend because it is not difficult to setup and avoid human mistake.

conda-forge is a different story: there are a few more steps and needs to be after. Once this is done, this is all automated.

[jordiferrero]

1. I don't know, but what would it be referencing then, if there is no release?
2. There is a bit of jargon... the changelog is created manually and is commited as any other code. It needs to be part of the tag/release.

The minimum workflow is as follow:

• have the code ready, including changelog
• make sure the version is set correctly
• make the sdist and wheel - there are different way to do it...
• upload to pypi manually

The two last steps can be done using github actions, which I would recommend because it is not difficult to setup and avoid human mistake.

conda-forge is a different story: there are a few more steps and needs to be after. Once this is done, this is all automated.

1. I wanted to simply get the base doi (concept idea) for the GitHub folder. I will see if Zenodo need a release to create a DOI. Otherwise I will set this up.

2. Can you create the GitHub actions for that, @ericpre ? Happy to learn too (but I am quite confused on how to start) so we can do that in the future...

[jlaehne]

I can also have a look at the actions, but never did it before either.

I created a developer team for the LumiSpy organization. You probably have to give that team some rights to the repository - so far I cannot even merge PRs and the like. We would also have to include @ericpre there in order that he could help out.

[jordiferrero]

I have now hopefully given the developer team full access to everything...

I have also linked Zenodo with the LumiSpy organisation.
As soon as we set up a first release, we will automatically get a DOI and a DOI badge.

I would propose for now to target the first v 0.1 release with the code as it is. All we should really add in the GitHub actions.
Or do you think there are any major issues we should cover first? Not sure how "perfect" of package it needs to be before we can have a release...

[jlaehne]

Great. As it is really just a v0.1 release, I would just go ahead to get it out.

If we use the github actions, further releases should not be so complicated and we can do another one as soon as any relevant new features are included.

[jordiferrero]

I agree.
So would you suggest to just go ahead as it is now?

And after that first v0.1 release we can set up the actions (aka set up pip install and conda-forge install.... which may take a bit of time)

[jlaehne]

I might at least give the actions a first try. Maybe I find some time to have a look in the next two days.

[ericpre]

Yes, after the first release, the installation instructions and README.md should be updated, for example to include the concept DOI, formatting of the README.md (pypi doesn't support all markdown), etc. A second release shortly after the first one will make sense to sort out all these details - it actually fairly easily to forget about these...

@jlaehne, if you want to explore the github actions, you can safely play with github actions in your own repository, make/delete tags when necessary.

It seems that the first upload to pypi needs to be done manually using twine: https://packaging.python.org/tutorials/packaging-projects/#next-steps, then we can create the token for the github actions.

[jordiferrero]

We have now our first release & a DOI (https://doi.org/10.5281/zenodo.4640445).

I may have made some mistakes, but I wanted to get the first release out so we can correct for them.

I have tried to install pip install lumispy to a clean conda environment and it works well!
Very happy!!

[jlaehne]

nice ;-)

[ericpre]

Indeed, this seems to be all good. I made a pull request to add the recipe to conda-forge, see conda-forge/staged-recipes#14406. You need to confirm that you are happy to be maintainer of the recipe and it will get reviewed and merged. Once this is done, the repository containing the feedstock will be created and you will have write access to that repository to maintain the recipe.

See conda-forge doc for more details:
https://conda-forge.org/docs/maintainer/adding_pkgs.html

[jlaehne]

Confirmed @conda.

I also added a PR to add LumiSpy to the hyperspy-extensions-list: hyperspy/hyperspy-extensions-list#6

[ericpre]

The conda-forge feedstock is now setup and the package works well: https://github.com/conda-forge/lumispy-feedstock.

Now that there is a conda-forge package, I will add it the hyperspy bundle for the next release of the bundle - most likely just after the next hyperspy release.

[jlaehne]

Started on the Github actions in #68

[jordiferrero]

Thank you all for this first release!