Add Linux and Mac arm platforms #352

AlbertDeFusco · 2022-01-11T06:22:00Z

This adds linux-aarch64 and osx-arm64 as known platforms.

Further, this PR also solves #274 and allows CONDA_SUBDIR env var to be used to override the current platform.

For example, on an Apple M1 you can now force it to create envs for osx-64. First you'll see that Conda determines my platform is osx-arm64. I can force anaconda-project to create an osx-64 env_spec by setting CONDA_SUBDIR=osx-64. Otherwise the prepare would fail because Python 3.7 is not available for osx-arm64. Note that you must keep CONDA_SUBDIR set for all anaconda-project actions, not just the prepare step.

> conda info --json | jq .platform
"osx-arm64"

> cat anaconda-project.yml
name: arm
packages:
- python=3.7

> anaconda-project prepare
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed

PackagesNotFoundError: The following packages are not available from current channels:

  - python=3.7

Current channels:

  - https://repo.anaconda.com/pkgs/main/osx-arm64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/osx-arm64
  - https://repo.anaconda.com/pkgs/r/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.


missing requirement to run this project: The project needs a Conda environment containing all required packages.
  '/Users/adefusco/Desktop/arm/envs/default' doesn't look like it contains a Conda environment yet.

> CONDA_SUBDIR=osx-64 anaconda-project prepare
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done

## Package Plan ##

  environment location: /Users/adefusco/Desktop/arm/envs/default

  added / updated specs:
    - python=3.7


The following NEW packages will be INSTALLED:

  ca-certificates    pkgs/main/osx-64::ca-certificates-2021.10.26-hecd8cb5_2
  certifi            pkgs/main/osx-64::certifi-2021.10.8-py37hecd8cb5_0
  libcxx             pkgs/main/osx-64::libcxx-12.0.0-h2f01273_0
  libffi             pkgs/main/osx-64::libffi-3.3-hb1e8313_2
  ncurses            pkgs/main/osx-64::ncurses-6.3-hca72f7f_2
  openssl            pkgs/main/osx-64::openssl-1.1.1l-h9ed2024_0
  pip                pkgs/main/osx-64::pip-21.2.2-py37hecd8cb5_0
  python             pkgs/main/osx-64::python-3.7.11-h88f2d9e_0
  readline           pkgs/main/osx-64::readline-8.1.2-hca72f7f_0
  setuptools         pkgs/main/osx-64::setuptools-58.0.4-py37hecd8cb5_0
  sqlite             pkgs/main/osx-64::sqlite-3.37.0-h707629a_0
  tk                 pkgs/main/osx-64::tk-8.6.11-h7bc2e8c_0
  wheel              pkgs/main/noarch::wheel-0.37.1-pyhd3eb1b0_0
  xz                 pkgs/main/osx-64::xz-5.2.5-h1de35cc_0
  zlib               pkgs/main/osx-64::zlib-1.2.11-h4dc903c_4


Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
#
# To activate this environment, use
#
#     $ conda activate /Users/adefusco/Desktop/arm/envs/default
#
# To deactivate an active environment, use
#
#     $ conda deactivate

The project is ready to run commands.
Use `anaconda-project list-commands` to see what's available.

* use CONDA_SUBDIR and CONDA_CHANNELS * adds osx-arm64 and linux-aarch64 platforms * uses Conda to determine the platform This last point is important espeically for Apple M1, which can run as the osx-64 platform. Users will now be able to force the platform by setting CONDA_SUBIDR=osx-64 before running anaconda-project commands.

if the .condarc file is not set then CONDA_CHANNELS does not know to include defaults

the msys2 channels has empty repodata for non-windows platforms

mcg1969

This looks great, but do we have a handle on those CI failures?

mcg1969 · 2022-01-14T03:59:14Z

By the way, I'd be open to dropping Python 2.7 support if necessary.

AlbertDeFusco · 2022-01-14T13:56:12Z

I'm going to keep working on the tests.

I may wait to drop 2.7 tests when I complete a PR to add py 3.9 testing. I think some tests fail when anaconda-project is installed in py 3.9 env and I'ld check if there are bugs.

--override-channels is added to address #336 some slow tests have been updated to avoid deadlocks identified in #360

the original version seemed to solve for python 2.7 on win-64. Hopefully this will fix windows tests in the action.

something didn't like python there

at least run more tests to find out why windows may be failing

.github/workflows/main.yml

there was no need to install the pip package multiple times also made sure to use modern python version

AlbertDeFusco · 2022-02-10T15:26:10Z

For those here who are interested I've gotten tests passing for all but three scenarios added the following exclusions to the github action.

The Ubuntu Conda 4.8 + py 3[7-8] fail for one test that checks a timestamp on the live project environment, but it does not exist for older or newer versions of Conda.

I plan to drop testing support for Py 2.7 in the near future.

          # Odd timestamp behavior
          - pyver: 3.7
            os: ubuntu-latest
            cver: 4.8
          - pyver: 3.8
            os: ubuntu-latest
            cver: 4.8
          # Old versions will be trimmed from tests soon
          - os: windows-latest
            pyver: 2.7
            cver: 4.6
          - os: windows-latest
            pyver: 3.6
            cver: 4.6

AlbertDeFusco · 2022-02-11T03:37:03Z

A question for those of you who maybe interested (@jbednar , @jlstevens , @philippjfr ?).

Anaconda Project had previously decided that the default platforms are linux-64, osx-64, and win-64. You see this when initializing a project or locking a project

platforms:
  - linux-64
  - osx-64
  - win-64

At some point osx-arm64 is going to start gaining in popularity, but right now only Python >=3.8 is available and many packages are missing so I may feel a little hesitant about adding the platform

Should osx-arm64 be added to the default platforms? Having osx-armt64 in default platform may cause locking to fail more often for users, and especially those who do not use osx-arm64.
If not should I attempt to contextualize the default platforms such that osx-arm64 is considered default only if anaconda-project is being run on osx-arm64? This would allow osx-arm64 to enter into the platforms: key without the user having to explicitly write it, as they would now and with this PR

AlbertDeFusco · 2022-02-11T03:44:39Z

oh, nevermind above! Anaconda Project already does option 2 above. I just need to adjust tests to account for this. Some tests were not originally intended to run on non default platforms hardware.

some quick fixes where paltforms was hardcoded or where conda_api.default_platforms_with_current() is a better choice in assertions.

found a scenario that works on osx-arm64

these tests had explicit platforms listed so it's ok to add some non-default platforms

some tests intentionall do not set platforms in the project yaml. If the test is run on a non-default platform, like osx-arm64 the hash is different because the platforms list is different. A mocked logical_hash is used when necessary to remove the platforms so that the test can pass on all hardware.

get CI tests passing again

AlbertDeFusco · 2022-02-13T14:35:24Z

Tests pass on all platforms. Any comments or concerns before I merge?

AlbertDeFusco added 8 commits September 1, 2021 10:21

forgot tqdm in dev env

4d7615d

VSCode settings, debug config, and devcontainer

509901e

explicit file name

3fa9dc2

fix kwargs for streaming_popen

1c0950a

fix non-x86 platform test

3b4151f

remove potential recursion

0ac08c8

fix one more test

23d408b

AlbertDeFusco requested a review from mcg1969 January 11, 2022 06:22

AlbertDeFusco added 9 commits January 11, 2022 07:32

lint

682707f

Merge branch 'test-fixes' into arm

847cacf

odd test behavior may be due to py 3.10

9afa9d3

don't forget the defaults channel

967a2a7

if the .condarc file is not set then CONDA_CHANNELS does not know to include defaults

matchspec typo

58bf595

it's ok to have msys2 in channels list

a257175

the msys2 channels has empty repodata for non-windows platforms

remove unused function

358e5f5

improve coverage

74708bb

lint

b03ce0b

mcg1969 reviewed Jan 14, 2022

View reviewed changes

AlbertDeFusco mentioned this pull request Feb 1, 2022

Allow one to use --override-channels for reproducibility. #336

Closed

AlbertDeFusco added 6 commits February 5, 2022 22:44

override channels and update slow tests

26b9c9e

--override-channels is added to address #336 some slow tests have been updated to avoid deadlocks identified in #360

update versions in slow test

43f5afe

the original version seemed to solve for python 2.7 on win-64. Hopefully this will fix windows tests in the action.

hmmm

11cdcb3

something didn't like python there

expand conda version support

1c36a0d

at least run more tests to find out why windows may be failing

versions are strings

4821c91

why was pytest pinned in github action?

f934b14

mcg1969 reviewed Feb 7, 2022

View reviewed changes

.github/workflows/main.yml Show resolved Hide resolved

try slightly older windows image on github

393bc9e

AlbertDeFusco added 5 commits February 8, 2022 21:55

maybe something odd happened to chardet

97c9933

need for info on failure

0d8b48d

oh, pip was missing!

39ee51a

refactor offending test on windows

1f0728d

there was no need to install the pip package multiple times also made sure to use modern python version

trim failed test combinations

a65f427

one more exclusion

be6eeaf

AlbertDeFusco requested a review from mcg1969 February 10, 2022 21:51

AlbertDeFusco added 10 commits February 10, 2022 23:32

osx-arm64 test fixes

23ea722

some quick fixes where paltforms was hardcoded or where conda_api.default_platforms_with_current() is a better choice in assertions.

real conda manager test

4201e1e

found a scenario that works on osx-arm64

add osx-arm64 where platforms provided

a0e5cff

these tests had explicit platforms listed so it's ok to add some non-default platforms

formatting fixes

c3f588e

update package versions for osx-arm64

68456c6

skip redis server tests if osx-arm64

591a22d

update versions for osx-arm64 support

cdbeb4b

avoid searching for site-packages for now

5d1b5d2

slight version adjustment

3cd280d

get CI tests passing again

This was referenced Feb 12, 2022

[BUG] tests fail with conda auto_stack #357

Open

Issue 337: Added pkg_key to ensure 'packages' == 'dependencies' #359

Open

AlbertDeFusco merged commit ca221d6 into master Feb 14, 2022

AlbertDeFusco mentioned this pull request Feb 14, 2022

rework the platform hack for modern conda #274

Closed

This was referenced Jul 8, 2022

[BUG] anaconda-project prepare doesn't recognize osx-arm64 platform #382

Closed

[BUG] failing tests with Py 3.9 #351

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Linux and Mac arm platforms #352

Add Linux and Mac arm platforms #352

AlbertDeFusco commented Jan 11, 2022

mcg1969 left a comment

mcg1969 commented Jan 14, 2022

AlbertDeFusco commented Jan 14, 2022

AlbertDeFusco commented Feb 10, 2022 •

edited

Loading

AlbertDeFusco commented Feb 11, 2022 •

edited by jbednar

Loading

AlbertDeFusco commented Feb 11, 2022 •

edited

Loading

AlbertDeFusco commented Feb 13, 2022 •

edited

Loading

Add Linux and Mac arm platforms #352

Add Linux and Mac arm platforms #352

Conversation

AlbertDeFusco commented Jan 11, 2022

mcg1969 left a comment

Choose a reason for hiding this comment

mcg1969 commented Jan 14, 2022

AlbertDeFusco commented Jan 14, 2022

AlbertDeFusco commented Feb 10, 2022 • edited Loading

AlbertDeFusco commented Feb 11, 2022 • edited by jbednar Loading

AlbertDeFusco commented Feb 11, 2022 • edited Loading

AlbertDeFusco commented Feb 13, 2022 • edited Loading

AlbertDeFusco commented Feb 10, 2022 •

edited

Loading

AlbertDeFusco commented Feb 11, 2022 •

edited by jbednar

Loading

AlbertDeFusco commented Feb 11, 2022 •

edited

Loading

AlbertDeFusco commented Feb 13, 2022 •

edited

Loading