Packaging and PyPI distribution?

[matthewfeickert]

As you've made a library and are using it as such:

# snippet from toy_distributions.ipynb

# append parent directory to `sys.path`
# to load from modules in `../l2hmc-qcd/`
module_path = os.path.join('..')
if module_path not in sys.path:
    sys.path.append(module_path)

# Local imports
from utils.attr_dict import AttrDict
from utils.training_utils import train_dynamics
from dynamics.config import DynamicsConfig
from dynamics.base_dynamics import BaseDynamics
from dynamics.generic_dynamics import GenericDynamics
from network.config import LearningRateConfig
from config import (State, NetWeights, MonteCarloStates,
                    BASE_DIR, BIN_DIR, TF_FLOAT)

from utils.distributions import (plot_samples2D, contour_potential,
                                 two_moons_potential, sin_potential,
                                 sin_potential1, sin_potential2)

do you have any plans and/or interest in packaging it as a Python library so it can either be pip installed from GitHub or be distributed on PyPI?

[saforem2]

Honestly it's something I've been meaning to do but just haven't gotten around to.

I'm not too familiar with the distribution side of packaging, so I'm not entirely sure what all it would require.

[matthewfeickert]

Honestly it's something I've been meaning to do but just haven't gotten around to.

Cool. 👍

I'm not too familiar with the distribution side of packaging, so I'm not entirely sure what all it would require.

If you want to talk sometime I'd be very happy to help/give some advice on this. The short list of recommendations is:

• Use a src directory layout

  • PythonBytes Episode #22 show notes: "Item 5: Python Package src-ery"
    • Full transcript of episode
    • pytest's recommendation for projects that have a __init__.py in the tests directory
    • Ionel Mărieș's argument in his "Packaging a python library" blog post on the subject
  • Hynek Schlawack's "Testing & Packaging" blog post
  • The PyPA's summary of their recommendations given their 2019 Packaging Mini Summit

• Use a CI/CD pipeline to do build tests

  • c.f. pyhf and recast-atlas for examples

• Use the PyPA's gh-action-pypi-publish GitHub Action to handle publishing to TestPyPI and PyPI

I'm happy to discuss this here, over email, or on a call.

[saforem2]

Sorry, I've been busy the past couple of days at this workshop but let me try getting into it over the weekend and see how it goes.

I'm happy to discuss this here, over email, or on a call.

Depending on how it goes, if I run into any issues or have any questions about anything I'll definitely make sure to take you up on this, thanks!

[matthewfeickert]

Nothing to apologize about. :)

Depending on how it goes, if I run into any issues or have any questions about anything I'll definitely make sure to take you up on this, thanks!

Cool. I'm happy to look over any PRs as well. 👍

[saforem2]

Bit delayed, but it's available on PyPI at l2hmc!

I still plan to setup CI / automated tests + releases so I'll keep this open for the time being as a (mostly as a reminder for myself)

Also I just wanted to say thanks for raising this in the first place!

Ultimately this led me to rewrite / restructure the base of the code to address the (many) issues that I had been stringing along with hacks and duct tape to that point, and I think the code is better as a result (plus I learned a lot!)