Packaging Python code with conda

Christian Fobel edited this page Aug 23, 2017 · 18 revisions

Table of Contents generated with DocToc

Most of the custom conda packages required for MicroDrop are hosted on the wheeler-mirofluidics and sci-bots channels. Starting with MicroDrop 2.8, plugins are also conda packages and they are hosted on their own microdrop-plugins channel.

Below are a few common commands that are useful for developing/building conda packages for MicroDrop.

Add channels

conda config --add channels conda-forge &
conda config --add channels wheeler-microfluidics &
conda config --add channels sci-bots

Show channels

conda config --show-sources

Install a local package in development mode

conda develop PATH_TO_PACKAGE

This works by creating a conda.pth file in site-packages.

Install build dependencies

conda install constructor conda-build git constructor-git nsis=3.0

Build a package

conda clean --source &
conda build .conda-recipe

By default, building a package using conda build requires an internet connection. See Building offline for instructions on how to build offline.

Note 1: Built packages are cached in the %CONDA_PREFIX%\conda-bld\win-32 directory.
Note 2: Running git clean --source is necessary to delete the git cache and prevent the package from being marked as a dev build (conda checks out a new branch if the cache exists). The cache is located at %CONDA_PREFIX%\conda-bld\git_cache\C_\Users\ryan\Documents\dev\python.

Errors building packages with dependencies with very long paths

Building packages that include very long file paths (I've mostly experienced issues with packages including platformio... packages) can fail with cryptic error messages, e.g.:

  • CondaFileNotFoundError
  • cannot access '...': No such file or directory

This may happen because, by default, the temporary Conda build environment is created as a sub-directory of the active Conda environment. Therefore, all paths within the build environment are necessarily longer than the corresponding paths in the root Conda environment.

For packages like, e.g., platformio-framework-arduinoavr, this can lead to very long paths, e.g.:

<conda-prefix>\<... conda build subdirectories ...>\pkgs\platformio-framework-arduinoavr-1.10617.4-0\share\platformio\packages\framework-arduinoavr\libraries\Adafruit_CircuitPlayground\examples\accelTap`

To work around this issue, you can use the --croot flag for the conda-build command, which allows you to specify an alternate parent directory for the temporary build environment. For example:

conda build --croot C:\cbld

Building offline

Unfortunately, there are a couple of issues that make building a Conda package offline tricky:

  1. conda-build does not directly provide an "offline" mode (see here and here).
  2. If the Conda build recipe uses a git repository as a source, building offline breaks (at least) the GIT_DESCRIBE_TAG and GIT_DESCRIBE__NUMBER Jinja template variables, causing conda-build to fail.

However, both of these issues can be worked around.

Workaround (1): Build using packages from local Conda channel repository

A local Conda channel repository, exposing all packages ever installed in the local Conda environment, can be built using the following steps:

  1. Activate Conda environment (e.g., activate).

  2. Change directory into %CONDA_PREFIX%\pkgs:

    cd %CONDA_PREFIX%\pkgs
    
  3. Index pkgs directory as Conda channel:

    conda index
    
  4. Create a directory to act as local Conda channel, e.g.:

    mkdir C:\pkg-cache
    
  5. Copy pkgs directory as win-32 in new package cache directory:

    xcopy /I C:\pkg-cache\win-32 %CONDA_PREFIX%\pkgs
    

    Note Using mklink seems to cause strange CondaFileNotFoundError errors when building. These errors do not seem to occur if the %CONDA_PREFIX% is actually copied (e.g., using xcopy ...) to the new package cache directory.

  6. Create an empty noarch directory in package cache directory (see here for info on why):

    mkdir C:\pkg-cache\noarch
    echo {} > C:\pkg-cache\noarch\repodata.json
    bzip2 -k C:\pkg-cache\noarch\repodata.json
    

Note The following command must be run to update the package index if new packages are installed into the Conda environment:

 cd %CONDA_PREFIX%\pkgs & conda index

Workaround (2): Use conda-render to prerender the build recipe before using conda-build

Building a Conda recipe with a git source using conda-build while offline seems to break the GIT_DESCRIBE_TAG and GIT_DESCRIBE__NUMBER, resulting in errors like the following:

ValueError: ("Fully-rendered version can't start with period -  got %s", '.0')

To work around this issue, the Conda build recipe may be pre-rendered as follows:

  1. Make a copy of the source Conda build recipe directory:

    xcopy /I .conda-recipe .conda-recipe-rendered
    
  2. Pre-render meta.yaml:

    conda render .conda-recipe > .conda-recipe-rendered\meta.yaml
    

The resulting .conda-recipe-rendered recipe can then be used as the recipe for the build command in the following section.

Build command

A package may be build using conda-build while being offline by specifying --override-channels and selecting a local channel that includes all required packages (see Workaround 1 above).

To build using the local channel, run:

conda build <recipe path> --override-channels -c file:///C:/pkg-cache

Note If building a recipe which uses a git repository as a source, the build recipe must be pre-rendered using conda-render (see Workaround 2 above).

Note Errors building packages with dependencies with very long paths may cause errors (see here).


Install package from local cache

You could use this, e.g., when you want to install a package that you have built using conda build .conda recipe.

conda install --use-local PACKAGE_NAME

Upload a package

anaconda upload -u wheeler-microfluidics XXX.tar.bz2

Build the MicroDrop installer (uses miniconda-recipe)

paver build_installer
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.