Migrate baseline images from DVC to Git LFS

[seisman]

The baseline images in pygmt/tests/baseline/*.png have been tracked using DVC since PR #1036 (five years ago; the original idea came from #963 (comment)). This setup has worked well, but it also has several drawbacks. I think it's time to consider whether we have a better solution.

Drawbacks of DVC

• Requires installing dvc, which is unfamiliar to most contributors
• The workflow is relatively complicated: dvc add, dvc push, git add, and git push (xref: https://www.pygmt.org/v0.19.0/contributing.html#using-data-version-control-dvc-to-manage-test-images)
• Every maintainer needs to configure DVC locally
• Contributors must be added as collaborators on DagsHub before they can push DVC-tracked files
• Pulling DVC-tracked files now requires authentication (Running "dvc pull" now requires authentication #4147)
• Several DVC-specific GitHub workflows are required to support the current setup

Git LFS

Git LFS is likely a better option for our use case. For files tracked by Git LFS, only small text pointer files are stored in the Git repository (similar to the current *.dvc files), while the actual file contents are stored separately. So updating baseline images won't increase the Git repository size.

Currently, PyGMT has ~240 baseline images with a total size of ~10 MB, which is well within GitHub LFS's free quota of 10 GB of storage and 10 GB of monthly bandwidth (xref: https://docs.github.com/en/billing/concepts/product-billing/git-lfs#free-use-of-git-lfs).

The setup is straightforward and only requires a few commands (see https://git-lfs.com/). Once configured, baseline images can be managed just like regular Git-tracked files using git add, git pull, and git push. This makes the workflow much more transparent and familiar for both maintainers and contributors.

Git LFS also integrates well with GitHub and pull requests. In my own fork, I have already migrated the baseline images to Git LFS on the main branch (https://github.com/seisman/pygmt). For example, if you visit https://github.com/seisman/pygmt/blob/main/pygmt/tests/baseline/test_basemap.png, GitHub displays the image directly, with a "Stored with Git LFS" note in the upper-left corner. When a baseline image is updated in a pull request, GitHub can also display image diffs directly (see https://github.com/seisman/pygmt/pull/4/changes for an example).

Overall, I think Git LFS satisfies our requirements while providing a simpler and more contributor-friendly workflow than DVC.

TODO

• Discuss whether to migrate from DVC to Git LFS
• Migrate pygmt/tests/baseline/*.png from DVC to Git LFS
• Remove dvc from environment.yml
• Remove .dvc/ and .dvcignore
• Update MANIFEST.in
• Update the Maintainers' Guide
• Update the Contributors' Guide
• Remove the dvc-diff.yml workflow, because GitHub can display diffs for LFS-tracked images directly in the web UI
• Remove the cache_dvc.yaml workflow, which was added as a workaround for Running "dvc pull" now requires authentication #4147
• Update the style_checks.yaml workflow
• Update the ci_tests.yaml workflow
• Update the ci_tests_dev.yaml workflow
• Update the release-baseline-image.yml workflow
• Ensure that source and binary distributions do not include the baseline images
• Remove DAGsHUB secrets from GitHub settings
• Decide if we want to delete the DagsHub repo https://dagshub.com/GenericMappingTools/pygmt, which is needed for old versions
• Explore if a similar migration make sense to the GMT repository

I have to admit that I’ve never used Git LFS before, so I’d especially like to hear from @weiji14, who implemented the DVC mechanism in PR #1036 and may have more experience with both DVC and Git LFS. I’m also pinging @GenericMappingTools/pygmt-maintainers for comments and feedback.

[Esteban82]

Would it be possible to migrate also the gmt repo?

[weiji14]

Thanks for re-assessing this @seisman, I'm re-reading my comment from 5 years ago at #963 (comment), and can't quite remember why we didn't deliberate on Git LFS before, except maybe...

Currently, PyGMT has ~240 baseline images with a total size of ~10 MB, which is well within GitHub LFS's free quota of 10 GB of storage and 10 GB of monthly bandwidth (xref: https://docs.github.com/en/billing/concepts/product-billing/git-lfs#free-use-of-git-lfs).

I think previously the limit was 1GB (according to https://stackoverflow.com/questions/51051553/how-to-reset-github-git-lfs-storage-limits), so a bit more dangerous. Note that 10MB is the current size, but Git LFS will need to store all the different diffs going forward (e.g. a 1MB file recreated will use up 2MB of space). 10GB sounds like it should be enough for a few years, given that A) we don't change the baseline images too often anymore, compared to a few years ago when things were in flux, and B) we do backups of the baseline images every version since #1317, so can always delete the older files in Git LFS somehow if needed to free up space (not sure how to do that, but possible in theory).

Git LFS also integrates well with GitHub and pull requests. In my own fork, I have already migrated the baseline images to Git LFS on the main branch (https://github.com/seisman/pygmt). For example, if you visit https://github.com/seisman/pygmt/blob/main/pygmt/tests/baseline/test_basemap.png, GitHub displays the image directly, with a "Stored with Git LFS" note in the upper-left corner. When a baseline image is updated in a pull request, GitHub can also display image diffs directly (see https://github.com/seisman/pygmt/pull/4/changes for an example).

Good to know that Diff previews work! Then we can also remove the dvc diff action (based on cml, which is slowly feeling less maintained...).

Overall, I think Git LFS satisfies our requirements while providing a simpler and more contributor-friendly workflow than DVC.

Yeah, I think LFS makes sense now. I don't use it very often, but would be comfortable with using it now. I did find my other comment at #451 (comment) to avoid git submodules (which I stand by, submodules are painful and a no no), but can't find a good reason to avoid LFS in the current state, other than the pain with migrating 🙂

[seisman]

Would it be possible to migrate also the gmt repo?

Yes, likely, but the baseline images in the GMT repo is ~200 MB, so it may face more challenges than PyGMT.

[yvonnefroehlich]

Sounds promising! I have no expericence with GitHub LFS yet, but I agree to change to it. The last two times I used DVC to update baseline images it caused problems, and I believe this DVC workflow may be also not directly clear to new contributors.