Pymatgen reorg

[shyuep]

I am starting a reorg of pymatgen. The first step is to create a pymatgen-core and pymatgen-test-files repo. These have been created under the materialyzeai organization (https://github.com/materialyzeai/pymatgen-core).

The idea is to retain "pymatgen" as a meta-package that includes pymatgen-core and all the bells and whistles. Pymatgen-core will just be the core packages, io, entries, etc. We will eventually move pymatgen-analysis and pymatgen-ext into their own packages.

pymatgen-test-files will be a separate Github repo that is a submodule of pymatgen-core and other first class packages. Here, we will use a submodule to avoid having to reclone all the huge test files. There is also a possibility that we make this a private submodule.

@DanielYang59 @mkhorton Let me know if you have any views and whether you want to be maintainers for the pymatgen-core and pymatgen-test-files repos.

[DanielYang59]

Hi thanks for asking me, sure I'm happy to help wherever I could

I think there would be benefit to break pymatgen into small packages but I'm a bit unsure about a dedicated pymatgen-test-files repo, which sounds like it beats the purpose of isolation (i.e. each package should be self contained to run tests, otherwise if someone want to modify/add a test he would need to make changes in both places)?

Also the test for core is quite small so I don't think it would beat the idea to make pymatgen-core lean:

 12K    alchemy/
 60M    analysis/
380K    apps/
388K    cif/
 29M    command_line/
212K    core/
143M    electronic_structure/
1.0M    entries/
313M    io/
 12K    performance/
4.3M    phonon/
 16M    surfaces/
 36K    symmetry/
 12K    transformations/

---

Before you spend too much time on the new pymatgen-core repo, can we start with splitting the repo with git-filter-repo to maintain the git history?

Also I think we could get some inspiration from how NOMAD is orchestrating a core nomad-fair package (the core part with infrastructure and so on) with around 100 plugins.

To the pmg side, I guess the current pymatgen package might become a wrapper to maintain backwards compatibility, something like:

[project]
name = "pymatgen"
dependencies = [
    "pymatgen-core==x",
    "pymatgen-io==y",
]

We might also turn pmg into a workspace repo:

pymatgen/
├── pyproject.toml              
├── docs/                     
├── pymatgen-core/          
├── pymatgen-io/     

[shyuep]

1. I already have a working version of pymatgen-core in the materialyzeai/pymatgen-core repo - I suggest you take a look and comment if there are any obvious things we should change (packages to include/exclude etc.)
2. The aims of the reorganization is two fold. The primary aim is to improve modularity and move mature code that is rarely modified to its own package. This has the added benefit of reducing cross-dependencies across packages (e.g., there are a few places that depend on pymatgen-analysis that would need to be moved). The secondary goal is to make the overhead smaller for codes that do not require the analysis parts.
3. Pymatgen-core does not mean just pymatgen/core package itself, but rather a suite of core packages. These include core, electronic_structure, io (at least some of it), transformations, etc. What is excluded is most of analysis, apps, alchemy, ext, etc. Some analysis modules that are really important (e.g., structure and molecular matcher, tensors, elasticity) will be moved to core.
4. One of the major things is backwards compatibility. We must ensure that pip install pymatgen retains all current functionality as is - no breaking changes. So there will need to be updates to stubs with proper deprecation messages etc.

The idea of having pymatgen-test-files in a separate repo (which is a submodule in the pymatgen-core repo and can be made the same in the pymatgen repo) is so that clones are not forced to download the entire suite of test files (which is very very large) unless necessary. The inconvenience is minimal since you simply add --recurse-submodules to git pull and you will still have access to all the test files. Adding test files to a submodule is also fairly simple.

As for NOMAD, we do not need to follow them. I have no intention of breaking pymatgen (at least the version in this repo) into 100 plugins. I am perfectly fine with just separating out the core packages for now. After that, we can decide what else can be moved to packages of their own. There are already many codes that depend on pymatgen. Some of these can depend on pymatgen-core in future if they do not need pymatgen/analysis.

Finally, for the git history, I will suggest we leave it in the pymatgen (non-core) repo for now. I don't think there is a need to inherit the git history to the pymatgen-core package.

[shyuep]

OK, @DanielYang59 @mkhorton I need to resolve this quickly. The longer this drags out, the more there are PR merges and the more difficult it becomes to keep things consistent.

1. Pymatgen-core is working and all tests pass. There was some reorg involved but nothing too major.
2. Pymatgen repo has been modified to use the pymatgen-test-files repo as a submodule. Most of the tests pass. The only ones that fail are due to the reorg in point 1 above and would be resolved very easily once I complete the move.

Next steps are simple. First I release pymatgen-core to PYPI. Second, I modify pymatgen main repo to remove all duplicate packages, add the necessary stubs for backwards compatibility, fix the tests and use pymatgen-core as a dependency.

Let me know by the end of this weekend if you have any objections before I proceed.

[esoteric-ephemera]

@shyuep: with @DanielYang59's help, we've put together a more complete rewrite of the pymatgen history. The reason for this is that a few core files were missed in the initial pass, because of various issues tracing history through renames

Method: Thanks to @DanielYang59 for putting this together - the main changes I made here were to not change the original committer in the `cherry-pick`:

#/bin/bash -l

if [ -d pymatgen-filtered ] ; then
    rm -rf pymatgen-filtered
fi

git clone https://github.com/materialsproject/pymatgen.git ./pymatgen-filtered && cd pymatgen-filtered

git filter-repo \
  --path-rename pymatgen/:src/pymatgen/ \
  --path src/pymatgen/alchemy/ \
  --path src/pymatgen/command_line/ \
  --path src/pymatgen/core/ \
  --path src/pymatgen/electronic_structure/ \
  --path src/pymatgen/io/ \
  --path src/pymatgen/optimization/ \
  --path src/pymatgen/phonon/ \
  --path src/pymatgen/transformations/ \
  --path src/pymatgen/symmetry/ \
  --path src/pymatgen/util/ \
  --path tests/ \
  --path .github/ \
  --path .gitignore \
  --path pyproject.toml \
  --path uv.lock \
  --path README.md \
  --path LICENSE \
  --path requirements.txt \
  --path setup.py \
  --path CITATION.cff \
  --path SECURITY.md \
  --path ADMIN.md \
  --force

git remote add origin https://github.com/esoteric-ephemera/pymatgen-core.git
git remote add upstream https://github.com/materialyzeai/pymatgen-core.git
git fetch --all

for hash in $(git rev-list --reverse 4ee64abde368..fa80531df413); do
  GIT_COMMITTER_NAME="$(git log -1 --format=%an $hash)" \
  GIT_COMMITTER_EMAIL="$(git log -1 --format=%ae $hash)" \
  GIT_COMMITTER_DATE="$(git log -1 --format=%ad $hash)" \
  git cherry-pick $hash -X theirs
done

for hash in $(git rev-list --reverse d1bd86715..6579eeb03) ; do
    GIT_COMMITTER_NAME="$(git log -1 --format=%an $hash)" \
    GIT_COMMITTER_EMAIL="$(git log -1 --format=%ae $hash)" \
    GIT_COMMITTER_DATE="$(git log -1 --format=%ad $hash)" \
    git cherry-pick -X theirs $hash
done

There are two routes is only one route I see for which this could be done:

• This branch would require force rebasing the history of pymatgen-core/main. You would have to do that directly @shyuep
• This branch could be merged directly via PR, but likely contains duplicate commits (which get relabeled when git filter-repo runs) Seemingly merging in these histories removes relevant history when there's conflict

[shyuep]

I suggest going the PR route. Alternatively, pls provide a script that I can use to run directly. Thanks.

[DanielYang59]

I believe we have to overwrite the main branch with force push (I don't think any other way without force push is possible)

Should be (can @esoteric-ephemera double check?):

git clone https://github.com/materialyzeai/pymatgen-core.git && cd pymatgen-core

# Add Aaron's fork 
git remote add source-repo https://github.com/esoteric-ephemera/pymatgen-core.git
git fetch source-repo

Then I believe you need to turn off the branch protection rule for now to allow force push to main:

git push origin source-repo/hist:main --force

then all forks need to do a hard reset (luckily currently there're only 4 forks, we should finish cleaning up the core repo as early as possible)

[shyuep]

Done.

[DanielYang59]

thanks, could confirm history are back as we expected, for example: https://github.com/materialyzeai/pymatgen-core/blame/main/src/pymatgen/core/structure.py

[esoteric-ephemera]

@shyuep I hear your concerns about time involved maintaining pymatgen and executing the split. However, I and others have fielded a lot of community concerns about authorship, organization, etc. that prevent them from wanting to contribute in the future

I propose that I and a few others work on preserving the split in the main repo, including git history, and ensuring that workflows for pymatgen-core and the full pymatgen run as needed, and deploy to pypi separately. We'll use the particular split you arrived at to guide the internal reorg. You would just need to ultimately review the PR.

Is this an acceptable solution? I will start immediately on this once you give the OK

[shyuep]

@JaGeo The code is ready and test files have been reintegrated into the repos. I suggest maybe your group take charge of moving the lobster and cohp over to pymatgen-core (should be a simple copy and paste) and then submit a PR (that way the code is attributed to your group). Once that is done, I will release a new pymatgen-core followed by the first release of pymatgen post-core extraction.

[JaGeo]

@naik-aakash and @tomdemeyere could you have a look how to do this in the best way?

[naik-aakash]

Small update, have added the code to pymatgen-core now materialsproject/pymatgen-core#5,

Once new release of pymatgen-core is made, I think the tests in the PR on pymatgen #4148 should also pass and could be merged as well.

[shyuep]

Thanks, I released pymatgen-core 2026.3.9. Does this mean we can delete all the io and electronic_structure classes?

[naik-aakash]

Yes, I had deleted them in my #4148 PR as well. I will resolve the merge conflicts in sometime