Support for partial clones

This release adds support for partial clones with mepo clone. A new option --partial has three settings:

• off: performs a "normal" git clone
• blobless: performs a blobless clone via git clone --filter=blob:none
• treeless: performs a treeless clone via git clone --filter=tree:0

The default is tecnically none of these, but that is equivalent to off. The off option is mainly added here to allow overriding of any .mepoconfig setting (see below).

The motivation for this is that it has been noticed that clones of GEOS (mainly MAPL) are often extremely slow. For example:

❯ time git clone git@github.com:GEOS-ESM/MAPL.git MAPL-normal
Cloning into 'MAPL-normal'...
remote: Enumerating objects: 704212, done.
remote: Counting objects: 100% (271489/271489), done.
remote: Compressing objects: 100% (6882/6882), done.
remote: Total 704212 (delta 269709), reused 266119 (delta 264584), pack-reused 432723
Receiving objects: 100% (704212/704212), 997.08 MiB | 2.59 MiB/s, done.
Resolving deltas: 100% (694344/694344), done.
noglob git clone git@github.com:GEOS-ESM/MAPL.git MAPL-normal  95.46s user 13.88s system 24% cpu 7:17.37 total

This took over 7 minutes to clone!

However, git supports partial clones as detailed in this GitHub Blog post. Now, of the two, blobless clones are fairly safe and give you faster initial clone speed at the cost of slower operations after that. As a test:

❯ time git clone --filter=blob:none git@github.com:GEOS-ESM/MAPL.git MAPL-blobless-from-git
Cloning into 'MAPL-blobless-from-git'...
remote: Enumerating objects: 21299, done.
remote: Counting objects: 100% (3171/3171), done.
remote: Compressing objects: 100% (1324/1324), done.
remote: Total 21299 (delta 1972), reused 2989 (delta 1829), pack-reused 18128
Receiving objects: 100% (21299/21299), 20.80 MiB | 2.99 MiB/s, done.
Resolving deltas: 100% (13343/13343), done.
remote: Enumerating objects: 942, done.
remote: Counting objects: 100% (552/552), done.
remote: Compressing objects: 100% (500/500), done.
remote: Total 942 (delta 112), reused 126 (delta 50), pack-reused 390
Receiving objects: 100% (942/942), 1.85 MiB | 136.00 KiB/s, done.
Resolving deltas: 100% (249/249), done.
Updating files: 100% (1064/1064), done.
noglob git clone --filter=blob:none git@github.com:GEOS-ESM/MAPL.git   1.38s user 0.54s system 6% cpu 27.609 total

28 seconds!

Treeless clones are usually faster than blobless as you aren't just filtering out blobs but whole trees. But per the blog:

We strongly recommend that developers do not use treeless clones for their daily work. Treeless clones are really only helpful for automated builds when you want to quickly clone, compile a project, then throw away the repository. In environments like GitHub Actions using public runners, you want to minimize your clone time so you can spend your machine time actually building your software! Treeless clones might be an excellent option for those environments.

As there are possible scenarios with CI that this could be useful for, the option is added. As for speed:

❯ time git clone --filter=tree:0 git@github.com:GEOS-ESM/MAPL.git MAPL-treeless
Cloning into 'MAPL-treeless'...
remote: Enumerating objects: 6875, done.
remote: Counting objects: 100% (843/843), done.
remote: Compressing objects: 100% (730/730), done.
remote: Total 6875 (delta 124), reused 805 (delta 113), pack-reused 6032
Receiving objects: 100% (6875/6875), 2.24 MiB | 277.00 KiB/s, done.
Resolving deltas: 100% (757/757), done.
remote: Enumerating objects: 106, done.
remote: Counting objects: 100% (70/70), done.
remote: Compressing objects: 100% (66/66), done.
remote: Total 106 (delta 1), reused 19 (delta 0), pack-reused 36
Receiving objects: 100% (106/106), 37.34 KiB | 538.00 KiB/s, done.
Resolving deltas: 100% (3/3), done.
remote: Enumerating objects: 942, done.
remote: Counting objects: 100% (552/552), done.
remote: Compressing objects: 100% (500/500), done.
remote: Total 942 (delta 112), reused 126 (delta 50), pack-reused 390
Receiving objects: 100% (942/942), 1.85 MiB | 2.09 MiB/s, done.
Resolving deltas: 100% (249/249), done.
Updating files: 100% (1064/1064), done.
noglob git clone --filter=tree:0 git@github.com:GEOS-ESM/MAPL.git   0.45s user 0.25s system 4% cpu 15.950 total

16 seconds!

Along with this option, we also add a new .mepoconfig setting where one can add:

[clone]
partial = blobless

and blobless clones will be the default.

---

From CHANGELOG.md

Added

• Added new --partial option to mepo clone with two settings: off, blobless, and treeless. If you set, --partial=blobless then
  the clone will not download blobs by using --filter=blob:none. If you set --partial=treeless then the clone will not download
  trees by using --filter=tree:0. The blobless option is useful for large repos that have a lot of binary files that you don't
  need. The treeless option is even more aggressive and SHOULD NOT be used unless you know what you are doing. The
  --partial=off option allows a user to override the default behavior of --partial in .mepoconfig and turn it off for a
  run of mepo clone.
• Add a new section for .mepoconfig to allow users to set --partial as a default for mepo clone.

---

What's Changed

• GitFlow: Merge main into develop for hotfix by @mathomp4 in #262
• Add support for partial clones by @mathomp4 in #264
• Protect submodule repos from treeless, add off option by @mathomp4 in #266
• GitFlow: Merge develop into main by @mathomp4 in #265

Full Changelog: v1.51.1...v1.52.0

Fix for ignore_submodules in older mepo clones

This patch release of mepo fixes an issue running mepo commands on older mepo clones made before v1.51.0 when ignore_submodules was added. Mea culpa.

---

From CHANGELOG.md

Fixed

• Fixes to allow mepo to work on older mepo clones that don't have ignore_submodules in their state

---

What's Changed

• Fixes for older mepo clones in re ignore_submodules by @mathomp4 in #261

Full Changelog: v1.51.0...v1.51.1

Add ability to ignore submodules in diff and status

TL;DR This release adds a new field to components.yaml, ignore_submodules which when true will ignore submodules in the mepo status and mepo diff calls.

---

A mepo oddity occurs in GEOSgcm because of how CICE and icepack are handled.

Technically, icepack is a submodule of CICE in the repo (see https://github.com/GEOS-ESM/CICE) and git sees it as that. However, currently GEOS handles icepack as a separate repo (most likely for ease of development) but git doesn't really know this. So when you go to CICE in GEOS in a fresh clone you see:

$ git status
HEAD detached at geos/v0.0.2
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   icepack (new commits)

Submodules changed but not updated:

* icepack 81a7628...8d80ff5 (1):
  > Merge pull request #1 from GEOS-ESM/bugfix/zhaobin74/recompute-enthalpy

no changes added to commit (use "git add" and/or "git commit -a")

and since mepo status just sort of "echoes" status output with mepo status we see:

cice6                  | (t) geos/v0.0.2 (DH)
   | icepack: modified, not staged

So git says "something changed" but we want to ignore that in this case.

Thus, this PR adds a new field to components.yaml, ignore_submodules:

cice6:
  local: ./src/Components/@GEOSgcm_GridComp/GEOSogcm_GridComp/GEOSseaice_GridComp/CICE_GEOSPlug/@cice6
  remote: ../CICE.git
  tag: geos/v0.0.2
  develop: geos/develop
  ignore_submodules: true

If this is true, then we add --ignore-submodules=all to underlying git calls when we run mepo diff and mepo status

---

From CHANGELOG.md

Added

• Added new ignore_submodules field in components.yaml to allow ignoring submodules in a repo. Currently used for status and
  diff commands.

---

What's Changed

• Add ability to ignore submodules by @mathomp4 in #258
• GitFlow: Merge Develop into main by @mathomp4 in #259

Full Changelog: v1.50.0...v1.51.0

Add hashes option to status command

This release of mepo adds a new --hashes option for mepo status that displays the hashes of each repo:

❯ mepo status
Checking status...
MAPL       | (b) main
ESMA_env   | (t) v4.19.0 (DH)
ESMA_cmake | (t) v3.31.0 (DH)
ecbuild    | (t) geos/v1.3.0 (DH)
❯ mepo status --hashes
Checking status...
MAPL       | (b) main (1afeb43d6af485c6fbedd7ce72ccc1eec20a5c5a)
ESMA_env   | (t) v4.19.0 (DH) (7a8e3b158967bf3144d396d836489f46888b7f56)
ESMA_cmake | (t) v3.31.0 (DH) (7dc4c819dc5169e6b887374aa0fa0a8f71846832)
ecbuild    | (t) geos/v1.3.0 (DH) (4a8c91a579e112a6e2a5224068e95c3a8935204b)

---

From CHANGELOG.md

Added

• Command status has now a --hashes option that list current HEAD hash for each component.

---

What's Changed

• Update zenodo.json by @mathomp4 in #252
• Add --hashes reading HEAD to get the current hash by @FlorianDeconinck in #253
• Update changelog for 1.50 release by @mathomp4 in #254
• GitFlow: Merge Develop into main for release by @mathomp4 in #255

New Contributors

• @FlorianDeconinck made their first contribution in #253

Full Changelog: v1.49.0...v1.50.0

Better mepo compare display

Previous to this release, if an original branch name is very long, this can cause odd output of mepo compare on narrow (relative to the branch name) terminals even if the repo wasn't changed. The compare command would use the maximum length of all repos, and not just those that changed.

With this fix, mepo compare will use only the repos that change to figure out how wide to make the output rather than all the branches on all the repos.

Of course, if you do mepo compare --all, it will base the output on all the repos as usual.

---

What's Changed

• Compare headers based on changed branch length by @mathomp4 in #249
• GitFlow: Merge Develop into Main for release by @mathomp4 in #250

Full Changelog: v1.48.0...v1.49.0

Add mepo reset command

This release adds a new mepo reset command:

❯ mepo reset -h
usage: mepo reset [-h] [-f] [--reclone] [-n]

Reset the current mepo clone to the original state.This will delete all subrepos and does not check for uncommitted changes! Must be run in the root of
the mepo clone.

options:
  -h, --help     show this help message and exit
  -f, --force    Force action.
  --reclone      Reclone repos after reset.
  -n, --dry-run  Dry-run only

By default, it will not re-clone a repo and will require a user to answer another question to run (without force flag):

❯ mepo clone
Initializing mepo using components.yaml
ESMA_env   | (t) v4.8.0
ESMA_cmake | (t) v3.21.0
ecbuild    | (t) geos/v1.3.0
❯ mepo reset
Are you sure you want to reset the project? If so, type 'yes' and press enter. yes
Removing ESMA_cmake/ecbuild...done.
Removing ESMA_cmake...done.
Removing ESMA_env...done.
Removing .mepo...done.
❯ ls -1 ESMA_env | head -1
ls: ESMA_env: No such file or directory

You can do a reclone with --reclone:

❯ mepo clone
Initializing mepo using components.yaml
ESMA_env   | (t) v4.8.0
ESMA_cmake | (t) v3.21.0
ecbuild    | (t) geos/v1.3.0
❯ mepo reset --reclone -f
Removing ESMA_cmake/ecbuild...done.
Removing ESMA_cmake...done.
Removing ESMA_env...done.
Removing .mepo...done.
Re-cloning all subrepos
Initializing mepo using components.yaml
ESMA_env   | (t) v4.8.0
ESMA_cmake | (t) v3.21.0
ecbuild    | (t) geos/v1.3.0
Recloning done.
❯ ls -1 ESMA_env | head -1
BASEDIR.rc.in

Note we also require the user to be in the root of a fixture:

❯ mepo clone
Initializing mepo using components.yaml
ESMA_env   | (t) v4.8.0
ESMA_cmake | (t) v3.21.0
ecbuild    | (t) geos/v1.3.0
❯ cd ESMA_cmake/ecbuild
❯ mepo reset
Error! As a safety precaution, you must be in the root directory of the project to reset

---

What's Changed

• Gitflow. Main to develop by @mathomp4 in #243
• Add new reset command by @mathomp4 in #245
• Update GitHub Actions by @mathomp4 in #247
• GitFlow: Merge Develop into Main For Release by @mathomp4 in #246

Full Changelog: v1.47.0...v1.48.0

Add mepo tag push --delete

This release of mepo adds a capability to delete tags on the remote with:

mepo tag push --delete tagname repo1 repo2

---

From CHANGELOG.md

Added

• Added ability to do mepo tag push --delete so you can delete a tag on the remote

---

What's Changed

• Add tag push delete by @mathomp4 in #240
• Merge Develop into Main for release by @mathomp4 in #241

Full Changelog: v1.46.0...v1.47.0

Add changed-files command

This release adds a new changed-files command to mepo. Essentially it will list all files that are different compared to the original tags/branches in the components.yaml (technically the mepo state, so if mepo save was run, the new components.yaml).

For example, if there are no changes:

❯ mepo compare
No repositories have changed
❯ mepo changed-files

Now we checkout a branch:

❯ mepo develop MAPL
Checking out development branch develop in MAPL
❯ mepo changed-files
./src/Shared/@MAPL/CHANGELOG.md
./src/Shared/@MAPL/generic/MAPL_Generic.F90

This shows that compared to where we started, we have two new files. Now let us alter a file and add a file:

❯ touch foo
❯ echo "bobob" >> src/Components/@GEOSgcm_GridComp/README.md
❯ mepo changed-files
./foo
./src/Shared/@MAPL/CHANGELOG.md
./src/Shared/@MAPL/generic/MAPL_Generic.F90
./src/Components/@GEOSgcm_GridComp/README.md

Finally, it has a --full-path option for a full path.

❯ mepo changed-files --full-path
/Users/mathomp4/Models/GEOSgcm-mepo-newfilesvorigin/GEOSgcm/foo
/Users/mathomp4/Models/GEOSgcm-mepo-newfilesvorigin/GEOSgcm/src/Shared/@MAPL/CHANGELOG.md
/Users/mathomp4/Models/GEOSgcm-mepo-newfilesvorigin/GEOSgcm/src/Shared/@MAPL/generic/MAPL_Generic.F90
/Users/mathomp4/Models/GEOSgcm-mepo-newfilesvorigin/GEOSgcm/src/Components/@GEOSgcm_GridComp/README.md

---

From CHANGELOG.md

Added

• Add new changed-files command to list all changed files vs original state

---

What's Changed

• Add ability to display changed files by @mathomp4 in #236
• GitFlow: Merge Develop into Main for release by @mathomp4 in #237

Full Changelog: v1.45.0...v1.46.0

Add tag support to checkout-if-exists

This release now allows for checkout-if-exists to work on tags or branches. For a branch:

❯ mepo checkout-if-exists feature/mathomp4/add-shared-baselibs-support -n
Branch feature/mathomp4/add-shared-baselibs-support exists in env
Branch feature/mathomp4/add-shared-baselibs-support exists in cmake
Branch feature/mathomp4/add-shared-baselibs-support exists in MAPL
Branch feature/mathomp4/add-shared-baselibs-support exists in GEOSgcm_GridComp

and for a tag:

❯ mepo checkout-if-exists sdr_r21c_v01 -n
Tag sdr_r21c_v01 exists in GEOSgcm_GridComp
Tag sdr_r21c_v01 exists in GEOSgcm_App

Before tags were just ignored.

---

From CHANGELOG.md

Changed

• Allow checkout-if-exists to work on tags or branches

---

What's Changed

• Allow checkout-if-exists to work on tags by @mathomp4 in #232
• Merge develop into main for release. by @mathomp4 in #233

Full Changelog: v1.44.0...v1.45.0

Add typechange support

This release adds support for typechange commits. At the moment, mepo status will say:

GOCART      | (b) main
   | ESMF/GOCART2G_GridComp/DU2G_GridComp/AMIP.20C/DU2G_instance_DU.rc: unknown (please contact mepo maintainer)

because it doesn't know about this. But with this releas we can now translate those:

GOCART      | (b) main
   | ESMF/GOCART2G_GridComp/DU2G_GridComp/AMIP.20C/DU2G_instance_DU.rc: typechange, staged but deleted, not staged

Note this might not be a complete support for typechange, but at least should cover the "main" commits.

---

From CHANGELOG.md

Fixed

• Add support for typechange in mepo status

---

What's Changed

• Fixes #228. Add typechange support by @mathomp4 in #229
• Merge Develop into Main for release by @mathomp4 in #230

Full Changelog: v1.43.0...v1.44.0