lincc-frameworks · drewoldag · Mar 13, 2023 · Mar 13, 2023 · Mar 13, 2023 · Mar 13, 2023
diff --git a/copier.yml b/copier.yml
@@ -79,6 +79,15 @@ include_notebooks:
         yes: true
         no: false
 
+use_gitlfs:
+    help: Do you want to add a .gitattributes with entries for git-lfs (see https://lincc-ppt.readthedocs.io/en/latest/practices/git-lfs.html)?
+    type: str
+    default: none
+    choices:
+        maybe later: none
+        yes .gitattributes with example entries disabled: disabled
+        yes .gitattributes with example entries enabled: enabled
+
 ###
 # Below this line are Copier configuration options.
 ###
@@ -88,6 +97,7 @@ _subdirectory: python-project-template
 
 _skip_if_exists:
     - README.md
+    - .gitattributes
 
 # Tasks to execute after generating or updating a project.
 _tasks:

diff --git a/docs/index.rst b/docs/index.rst
@@ -25,6 +25,7 @@ It's really neat!
    practices/overview
    practices/ci_testing
    practices/code_coverage
+   practices/git-lfs
    practices/linting
    practices/precommit
    practices/pypi

diff --git a/docs/practices/git-lfs.rst b/docs/practices/git-lfs.rst
@@ -0,0 +1,116 @@
+Git Large File Support
+===============================================================================
+
+Quick Start
+-------------------------------------------------------------------------------
+
+After cloning a repository and connecting to the resulting working tree confirm that you have ``git-lfs`` and then issue ``git lfs install`` to install configuration and hooks and then ``git lfs track`` to designate targets for git-lfs.
+
+.. code-block:: bash
+
+    git-lfs --version
+    git lfs install --local
+    git lfs track '*.fits' '*.fits.fz' '*.gz'
+
+
+.. list-table::
+  :header-rows: 1
+
+  * - **Common Problems**
+    - **Resolution**
+  * - git-lfs: command not found
+    - Install git-lfs, see :ref:`get-git-lfs-label`
+  * - git: 'lfs' is not a git command
+    - Install git-lfs, see :ref:`get-git-lfs-label`
+
+What is it? Why do it?
+-------------------------------------------------------------------------------
+
+`Git-lfs <https://git-lfs.com/>`_ replaces large files such as datasets, and graphics with text pointers inside Git, while storing the file contents on a remote server.
+This can be very useful for projects that have large data files that change infrequently. It does require a remote that supports git-lfs and so if you are unsure about
+whether you want to use git-lfs you probably do not want to use it until you understand it better.
+
+This template provides a starting point for using git-lfs with a project.
+Note that you need to install the program ``git-lfs`` separately as that is not easily done as part of the installation. See :ref:`get-git-lfs-label`.
+
+How to manage
+-------------------------------------------------------------------------------
+
+.. _get-git-lfs-label:
+
+Get git-lfs
+^^^^^^^^^^^
+
+The preferred installation instructions are at `Git-lfs <https://git-lfs.com/>`_.
+That site will show you instructions most appropriate for your platform.
+
+You may also use conda as it has a ``git-lfs`` package appropriately named.
+
+.. code-block:: bash
+
+    conda install git-lfs
+
+At this time it is not available using ``pip``  or ``pipx``.
+Note that while there is a pip package named ``git-lfs`` it does not in fact contain
+``git-lfs``. Please do not use it.
+
+Install git-lfs in the local repository
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Once you have git-lfs use it to install support into the repository with the command
+
+.. code-block:: bash
+
+    git lfs install --local
+
+The option ``--local`` is necessary only if you want to avoid modifications to .gitconfig in
+your home directory. Quite likely you do not care and so may omit the option. Note that you must
+run the command for each repository requiring git-lfs support and you must run it after each
+clone because it installs hooks in .git/hooks.
+
+You will need to use a remote that supports git-lfs. Git-hub, the default for LINCC Frameworks,
+and several other git servers do support git-lfs. If you use another remote you should check.
+
+Designate files for git-lfs tracking
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+You will also want add paths to ``.gitattributes`` that designate the files git-lfs will manage for git.
+While you can edit ``.gitattributes`` directly there is a command that will make the needed changes and create the file if necessary.
+For example, if you want to specify that ``FITS`` files are handled by git-lfs then you might use these commands
+
+.. code-block:: bash
+
+    git lfs track '*.fits' '*.fits.fz'
+
+You can see a list of currently tracked names
+
+.. code-block:: bash
+
+    git lfs track
+
+See ``Getting Started`` at `Git-lfs <https://git-lfs.com/>`_ for more details.
+
+We also recommend you review `Using Git LFS-enabled repositories <https://developer.lsst.io/git/git-lfs.html#using-git-lfs-enabled-repositories>`_
+in the lsst developer documentation and adapt its recommendations to your project.
+
+Uninstall git-lfs from the repository
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If you want to stop using git-lfs going forward then you can uninstall it. You should use the same options used on installation, in particular if you specified the ``--local`` option you should also specify it here. Probably it is safer to always include the ``--local`` option so you restrict impact to the current repository.
+
+.. code-block:: bash
+
+    git lfs uninstall --local
+
+Any files modified after uninstalling git-lfs will become part of the regular git repository but the git-lfs artifacts will remain.
+
+Note that we recommend against uninstalling git-lfs as it causes confusion.
+
+You can see what files are controlled by git-lfs using the ``git lfs ls-files`` command
+
+.. code-block:: bash
+
+    git lfs ls-files
+
+Then if you want to copy them to the regular git repository you can change their modification dates using ``touch`` and commit the changes.
+
+
+
diff --git a/docs/practices/overview.rst b/docs/practices/overview.rst
@@ -13,4 +13,5 @@ Practices
 * :doc:`linting`
 * :doc:`precommit`
 * :doc:`pypi`
-* :doc:`sphinx`
+* :doc:`sphinx`
+* :doc:`git-lfs`
diff --git a/docs/source/new_project.rst b/docs/source/new_project.rst
@@ -36,6 +36,8 @@ Copier will ask you questions for how to set up the project. These questions wil
      - `isort <https://pycqa.github.io/isort/>`_ is a tool for ordering imports in a standard order. Enabling the option will include ``isort`` as part of github's :doc:`pre-commit <../practices/precommit>`. Defaults to ``True`` during simple installation.
    * - *Do you want to create some example module code?*
      - If this option is selected the template will create a model in ``src/{{module_name}}`` and create a corresponding example test file. Defaults to ``True`` during simple installation.
+   * - *Do you want to add a .gitattributes with entries for git-lfs?*
+     - Support for large files for use in git. This option is primarily informational and no answer locks you in to using (or not using) git-lfs. Importantly, selecting this option does not install git-lfs for your project (see :doc:`Git_Large_File_Support <../practices/git-lfs>`).
 
 While these choices will provide the initial structure for your project, most can be changed later. See Copier's `documentation for changing answers to the question <https://copier.readthedocs.io/en/stable/updating/>`_ 
 
@@ -71,8 +73,8 @@ Commit your new project locally
 -------------------------------------------------------------------------------
 
 If you're interested in using pre-commit hooks to crosscheck your code before you commit it, 
-now is a good time to set that up (it's just one command) - check out 
-:doc:`</practices/precommit>`.
+now is a good time to set that up (it's just one command) - check out
+:doc:`pre-commit <../practices/precommit>`
 
 Commit the project to your local version control like so to see the pre-commit checks run.
 
@@ -108,4 +110,4 @@ Yep. That's it.
 
 Copier will automatically check to see if a newer version of the original template is available and if so the changes will be automatically applied. Neato!
 
-And of course, because your project is under version control, if you don't like the new changes, you can always revert back to the previous state.
+And of course, because your project is under version control, if you don't like the new changes, you can always revert back to the previous state.
diff --git a/python-project-template/.gitignore b/python-project-template/.gitignore
@@ -134,4 +134,7 @@ dmypy.json
 .vscode/
 
 # dask
-dask-worker-space/
+dask-worker-space/
+
+# tmp directory
+tmp/
diff --git a/python-project-template/{% if use_gitlfs == 'disabled' %}.gitattributes{% endif %}.jinja b/python-project-template/{% if use_gitlfs == 'disabled' %}.gitattributes{% endif %}.jinja
@@ -0,0 +1,21 @@
+# For explanation of this file and uses see
+# https://git-scm.com/docs/gitattributes
+# https://developer.lsst.io/git/git-lfs.html#using-git-lfs-enabled-repositories
+#
+# Used by https://github.com/lsst/afwdata.git
+# *.boost filter=lfs diff=lfs merge=lfs -text
+# *.dat filter=lfs diff=lfs merge=lfs -text
+# *.fits filter=lfs diff=lfs merge=lfs -text
+# *.gz filter=lfs diff=lfs merge=lfs -text
+#
+# apache parquet files
+# *.parq	filter=lfs diff=lfs merge=lfs -text
+#
+# sqlite files
+# *.sqlite3	filter=lfs diff=lfs merge=lfs -text
+#
+# gzip files
+# *.gz	filter=lfs diff=lfs merge=lfs -text
+#
+# png image files
+# *.png	filter=lfs diff=lfs merge=lfs -text
diff --git a/python-project-template/{% if use_gitlfs == 'enabled' %}.gitattributes{% endif %}.jinja b/python-project-template/{% if use_gitlfs == 'enabled' %}.gitattributes{% endif %}.jinja
@@ -0,0 +1,21 @@
+# For explanation of this file and uses see
+# https://git-scm.com/docs/gitattributes
+# https://developer.lsst.io/git/git-lfs.html#using-git-lfs-enabled-repositories
+#
+# Used by https://github.com/lsst/afwdata.git
+*.boost filter=lfs diff=lfs merge=lfs -text
+*.dat filter=lfs diff=lfs merge=lfs -text
+*.fits filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+#
+# apache parquet files
+*.parq	filter=lfs diff=lfs merge=lfs -text
+#
+# sqlite files
+*.sqlite3	filter=lfs diff=lfs merge=lfs -text
+#
+# gzip files
+*.gz	filter=lfs diff=lfs merge=lfs -text
+#
+# png image files
+*.png	filter=lfs diff=lfs merge=lfs -text