Skip to content

Commit

Permalink
Merge branch 'latest' into fix_max_hash_deprecations
Browse files Browse the repository at this point in the history
  • Loading branch information
luizirber committed Feb 5, 2021
2 parents 57bd8cf + 3c1241b commit b821f76
Show file tree
Hide file tree
Showing 15 changed files with 441 additions and 93 deletions.
64 changes: 64 additions & 0 deletions .github/workflows/dev_envs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
name: "Dev env instructions"
on:
pull_request:
branches: [latest]
push:
branches: [latest]
jobs:
nix:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2.3.4
with:
fetch-depth: 0

- name: Cache nix store
id: cache-nix
uses: actions/cache@v2
with:
path: /nix/store
key: nix-${{ hashFiles('shell.nix') }}-${{ hashFiles('nix/**') }}

- uses: cachix/install-nix-action@v12
with:
nix_path: nixpkgs=channel:nixos-20.09

- run: nix-shell --command "tox -e py39"

mamba:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2.3.4
with:
fetch-depth: 0

- name: cache conda
uses: actions/cache@v1
env:
CACHE_NUMBER: 0
with:
path: ~/conda_pkgs_dir
key:
${{ runner.os }}-conda-${{ env.CACHE_NUMBER }}-${{ hashFiles('environment.yml') }}

- name: setup conda
uses: conda-incubator/setup-miniconda@e23d871804685e8c52189e5bd45e9145019f10af
with:
auto-update-conda: true
python-version: 3.9
channels: conda-forge,bioconda
miniforge-variant: Mambaforge
miniforge-version: latest
use-mamba: true
mamba-version: "*"
activate-environment: sourmash_dev
auto-activate-base: false
use-only-tar-bz2: true

- name: install dependencies
shell: bash -l {0}
run: mamba install tox-conda rust git compilers pandoc

- name: run tests for 3.9
shell: bash -l {0}
run: tox -e py39
88 changes: 68 additions & 20 deletions doc/developer.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,43 +6,79 @@ You can get the latest development branch with:
```
git clone https://github.com/dib-lab/sourmash.git
```
sourmash runs under Python 3.7 and later. The base
requirements are screed and cffi, together with a Rust environment (for the
extension code). We suggest using `rustup` to install the Rust environment:
sourmash runs under Python 3.7 and later.

curl https://sh.rustup.rs -sSf | sh
We recommend using `conda` or `Nix` for setting up an environment for developing
new features, running tests and code quality checks.
Here are some suggestions on how to set them up (note: you only need one =])

We use [`tox`](https://tox.readthedocs.io) for managing dependencies and
running tests and checks during development.
To install it, do:
### Using mamba (conda alternative)

Follow the [installation instructions](https://github.com/conda-forge/miniforge#install) for
installing `mambaforge` (a conda distribution that uses
[`mamba`](https://github.com/TheSnakePit/mamba)
and the [`conda-forge`](https://conda-forge.org/) channel by default).

Once `mamba` is installed, run
```
mamba create -n sourmash_dev tox-conda rust git compilers pandoc
```
to create an environment called `sourmash_dev` containing the programs needed
for development.

To activate the new environment, run
```
conda activate sourmash_dev
```
and proceed to the ["Running tests and checks"](#running-tests-and-checks) section.

### Using Nix

Follow the [installation instructions](https://nixos.org/manual/nix/stable/#chap-installation)
for setting up Nix in your system (Linux or macOS).

Once Nix is installed, run
```
nix-shell
```
to start an environment ready for [running tests and checks](#running-tests-and-checks).

### General instructions

As long as you have `tox` and a Rust compiler available,
you can skip `mamba` or `Nix`.

For Rust, we suggest using `rustup` to install the Rust environment:
```
curl https://sh.rustup.rs -sSf | sh
```
And for `tox` you can run
```
python -m pip install tox
```
and use `tox -l` to list available tasks.

We suggest working on sourmash in a virtualenv; e.g. from within the
sourmash clone directory, you can do:
cloned repository (and after installing `tox` and Rust), you can do:
```
tox -e dev
. .tox/dev/bin/activate
```

You can run tests by invoking `make test` in the sourmash directory;
`tox -e py39` will run the Python tests with Python 3.9,
and `cargo test` will run the Rust tests.

You can also explicitly install all the dependencies for sourmash by running
Finally, ou can also explicitly install all the Python dependencies for sourmash by running
```
pip install -r requirements.txt
```
(but they are already installed in the virtualenv created with `tox -e dev`).

### If you're having trouble installing or using the development environment
## Running tests and checks

If you are getting an error that contains `ImportError: cannot import name 'to_bytes' from 'sourmash.minhash'`, then it's likely you need to update Rust and clean up your environment. Some installation issues can be solved by simply removing the intermediate build files with:
We use [`tox`](https://tox.readthedocs.io) for managing dependencies and
running tests and checks during development.
`tox -l` lists available tasks.

```
make clean
```
You can run tests by invoking `make test` in the sourmash directory;
`tox -e py39` will run the Python tests with Python 3.9,
and `cargo test` will run the Rust tests.

## Adding new changes

Expand Down Expand Up @@ -99,7 +135,7 @@ A short description of the high-level files and dirs in the sourmash repo:
├── Makefile | Entry point for most development tasks
├── MANIFEST.in | Describes what files to add to the Python package
├── matplotlibrc | Configuration for matplotlib
├── nix.shell | Nix configuration for creating a dev environment
├── shell.nix | Nix configuration for creating a dev environment
├── paper.bib | References in the JOSS paper
├── paper.md | JOSS paper content
├── pyproject.toml | Python project definitions (build system and tooling)
Expand Down Expand Up @@ -218,6 +254,18 @@ For the Rust core library we use `rMAJOR.MINOR.PATCH`
The Rust version is not automated,
and must be bumped in `src/core/Cargo.toml`.

## Common errors and solutions

### Cannot import name `to_bytes` from `sourmash.minhash`

If you are getting an error that contains `ImportError: cannot import name 'to_bytes' from 'sourmash.minhash'`,
then it's likely you need to update Rust and clean up your environment.
Some installation issues can be solved by simply removing the intermediate build files with:

```
make clean
```

## Contents

```{toctree}
Expand Down
51 changes: 0 additions & 51 deletions nix.shell

This file was deleted.

38 changes: 38 additions & 0 deletions nix/sources.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
{
"niv": {
"branch": "master",
"description": "Easy dependency management for Nix projects",
"homepage": "https://github.com/nmattia/niv",
"owner": "nmattia",
"repo": "niv",
"rev": "3cd7914b2c4cff48927e11c216dadfab7d903fe5",
"sha256": "1agq4nvbhrylf2s77kb4xhh9k7xcwdwggq764k4jgsbs70py8cw3",
"type": "tarball",
"url": "https://github.com/nmattia/niv/archive/3cd7914b2c4cff48927e11c216dadfab7d903fe5.tar.gz",
"url_template": "https://github.com/<owner>/<repo>/archive/<rev>.tar.gz"
},
"nixpkgs": {
"branch": "nixpkgs-unstable",
"description": "Nix Packages collection",
"homepage": "",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "d1f97a5eb5115289071d8449f26e7b92ce6b6709",
"sha256": "098rb747w4p5lxz74bj738bpjd4xjn0ahssjp9n8237dmmx3wg5p",
"type": "tarball",
"url": "https://github.com/NixOS/nixpkgs/archive/d1f97a5eb5115289071d8449f26e7b92ce6b6709.tar.gz",
"url_template": "https://github.com/<owner>/<repo>/archive/<rev>.tar.gz"
},
"rust-overlay": {
"branch": "master",
"description": null,
"homepage": null,
"owner": "oxalica",
"repo": "rust-overlay",
"rev": "0bb9ef6d8b34e5579d7384201f3106a49ce3deca",
"sha256": "09cva1r53x1ihyzbssdscx4n5913pra8f006q095ww9wvfvz8bxf",
"type": "tarball",
"url": "https://github.com/oxalica/rust-overlay/archive/0bb9ef6d8b34e5579d7384201f3106a49ce3deca.tar.gz",
"url_template": "https://github.com/<owner>/<repo>/archive/<rev>.tar.gz"
}
}
Loading

0 comments on commit b821f76

Please sign in to comment.