Port to a new CPU ISA

[willliu]

• I read the conda-forge documentation and could not find the solution for my problem there.

Issue: Dear conda community, we recently encountered a new CPU ISA called loongarch64, and would like to port conda forge to this new CPU ISA.

Here is what we have attempted:

1. check miniforge build script
2. check docker-images build file

Then we found the following issues to prevent us to use the existing scripts:

1. We don't yet have a centos ready, but could potentially prepare a debian based docker images (with qemu tools installed)
2. docker-images's scripts/run_commands depends on the miniforge download file, which is not available for this new ISA

Now, we are wondering whether the community could generously point a guidance to help us get started.
Btw, of course, we have a working native machine running debian based linux with kernel 4.19 and all development tools.

Many thanks,

Environment (conda list): Not available

$ conda list

Details about conda and system ( conda info ): Not available

$ conda info

[isuruf]

Here are roughly the steps you need to follow in order,

1. Add support to https://github.com/conda/conda
2. Add support to https://github.com/conda/conda-build
3. Ask for support in https://anaconda.org
4. Add support in https://github.com/conda-forge/linux-sysroot-feedstock
5. Add support for cross compiling in https://github.com/conda-forge/binutils-feedstock
6. Add support for cross compiling in https://github.com/conda-forge/ctng-compilers-feedstock
7. Add support for cross compiling in https://github.com/conda-forge/ctng-compiler-activation-feedstock
8. Add support for QEMU in https://github.com/conda-forge/docker-images
9. Add support for native compiling in https://github.com/conda-forge/binutils-feedstock
10. Add support for native compiling in https://github.com/conda-forge/ctng-compilers-feedstock
11. Add support for native compiling in https://github.com/conda-forge/ctng-compiler-activation-feedstock
12. Cross compile all dependencies of conda-build and micromamba
13. Build a miniforge installer

[willliu]

@isuruf Thank you (and take a leave to check the list for now)

[willliu]

@isuruf Mr Fernando, after a few days, Conda is managed to be built and installed. May I ask two questions.

Question 1:

When install conda form source via sudo python3 install setup.py, I have to comment out the requirements of ruamel_yaml_conda in setup.py and then use sudo pip3 install ruamel.yaml to make sure conda works. Should this be changed? Or that I have to somehow provide ruamel_yaml_conda recipe for loongarch64 before conda can be upstreamed?

Question 2:

The command conda info gives the arch linux-64, and conda init does nothing, and conda list requires conda to initialize. I guess these are pointing the places that I need to start to modify, especially the CPU arch related sections in the source code, are I right?

$conda info

     active environment : None
       user config file : /home/xxx/.condarc
 populated config files : 
          conda version : 4.10.3.post10+8bf56feb7
    conda-build version : 3.21.5
         python version : 3.7.3.final.0
       virtual packages : __linux=4.19.0=0
                          __glibc=2.28=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /usr  (read only)
      conda av data dir : /usr/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /usr/pkgs
                          /home/xxx/.conda/pkgs
       envs directories : /home/xxx/.conda/envs
                          /usr/envs
               platform : linux-64
             user-agent : conda/4.10.3.post10+8bf56feb7 requests/2.21.0 CPython/3.7.3 Linux/4.19.0-12-loongson-3 loongnix/20 glibc/2.28
                UID:GID : 1000:1001
             netrc file : None
           offline mode : False

$sudo conda init

no change     /usr/condabin/conda
no change     /usr/bin/conda
no change     /usr/bin/conda-env
no change     /usr/bin/activate
no change     /usr/bin/deactivate
no change     /usr/etc/profile.d/conda.sh
no change     /usr/etc/fish/conf.d/conda.fish
no change     /usr/shell/condabin/Conda.psm1
no change     /usr/shell/condabin/conda-hook.ps1
no change     /usr/lib/python2.7/site-packages/xontrib/conda.xsh
no change     /usr/etc/profile.d/conda.csh
no change     /usr/condabin/conda
no change     /usr/bin/conda
no change     /usr/bin/conda-env
no change     /usr/bin/activate
no change     /usr/bin/deactivate
no change     /usr/etc/profile.d/conda.sh
no change     /usr/etc/fish/conf.d/conda.fish
no change     /usr/shell/condabin/Conda.psm1
no change     /usr/shell/condabin/conda-hook.ps1
no change     /usr/lib/python2.7/site-packages/xontrib/conda.xsh
no change     /usr/etc/profile.d/conda.csh
no change     /root/.bashrc
No action taken.

$conda list

CondaError: Conda has not been initialized.

To enable full conda functionality, please run 'conda init'.
For additional information, see 'conda init --help'.

Great appreciation for your patience.

[isuruf]

I suggest you first install conda on an existing platform like linux-x86_64 and do the changes necessary and all the changes upto step 13 on linux-x86_64 or similar. After step 13, you'll be able to run conda on loongarch64.

[willliu]

@isuruf OK

[willliu]

@isuruf one quick question, to support loongarch64, does it rely on a native Travis-Cl machine with loongarch64, like mentioned here s390 conda support

[isuruf]

No, you can cross compile using conda-smithy and conda-build infrastructure and run test programs using qemu.

[willliu]

@isuruf Thank you for your clarification.

[loongson-zn]

@willliu discuss it, send me an e-mail (six_na at 163.com).

[liuxiang88]

@isuruf hi,there has two new quesions for setup 3:
1 How to contract with anaconda?I had submitted on https://www.anaconda.com/contact at 2022.4，but no response.So where should I go to contact them?
2 What kind of support should I ask them to provide？Should I ask them to provide a sh file containing elf?

Many thanks,

[isuruf]

As I said before, you need to do these things in order. First you need to support the subdir linux-loongarch64 in conda and conda-build.
Then for step 3, you can ask @chenghlee or @jezdez to allow upload of linux-loongarch64 packages to be uploaded to anaconda.org, but you need the first and second step to happen before that can be done.

[willliu]

@isuruf Just get reminders and check back.

Last year when porting the conda-build, I found being in an awkward situation of missing a CentOS host for loongarch64. Now @loongson-zn has came in touch, and I believed who will handle this issue to make conda/conda-build patched. ^_^

[loongson-zn]

yeah，we are actively pursuing this work and will contact anaconda's friends as soon as possible. Now we have both debian and centos systems, conda and conda-build adaptation has been completed.

@isuruf Just get reminders and check back.

Last year when porting the conda-build, I found being in an awkward situation of missing a CentOS host for loongarch64. Now @loongson-zn has came in touch, and I believed who will handle this issue to make conda/conda-build patched. ^_^

[liuxiang88]

@isuruf Thank you for your reply.Conda has been submitted and conda-build is checking code.
@chenghlee hi,I have two quesions about the conda-forge repositories:
1 What is the process required to build the conda-forge warehouse of loongarch64 branch directory and what do I need to do?
2 I want to build a local repositories for testing before submitting it to anaconda/conda-forge. Is there a manual for local repositories construction that can be provided?

[hmaarrfk]

Steps 12 and 13 are the manual steps you speak of.

For a while, I had my own project Archiconda https://github.com/Archiconda that I build in total isolation from conda-forge (with help from others here interested to get the critical infrastructure).

I had packages uploaded to Anaconda as well https://anaconda.org/archiarm

You can do ALOT to prove out the feasibility of conda for your custom ISA without relying on other's for their PRs and acceptance. Since 2018, I think https://github.com/mamba-org/quetz has come up, and you could likely even host your own server before Anaconda moves on your requests.

Then, eventually, we can build out a long term, opt-in migrator, as we did for aarch64 + ppc64le.

[hmaarrfk]

Then, eventually, we can build out a long term, opt-in migrator, as we did for aarch64 + ppc64le.

We likely won't do this until all the other steps are moved through, and that you've proved out a few of the basics through your manual process, as you mentionned.

I hope the reference to the other tools helps.

[loongson-zn]

Here are roughly the steps you need to follow in order,

1. Add support to https://github.com/conda/conda
2. Add support to https://github.com/conda/conda-build
3. Ask for support in https://anaconda.org
4. Add support in https://github.com/conda-forge/linux-sysroot-feedstock
5. Add support for cross compiling in https://github.com/conda-forge/binutils-feedstock
6. Add support for cross compiling in https://github.com/conda-forge/ctng-compilers-feedstock
7. Add support for cross compiling in https://github.com/conda-forge/ctng-compiler-activation-feedstock
8. Add support for QEMU in https://github.com/conda-forge/docker-images
9. Add support for native compiling in https://github.com/conda-forge/binutils-feedstock
10. Add support for native compiling in https://github.com/conda-forge/ctng-compilers-feedstock
11. Add support for native compiling in https://github.com/conda-forge/ctng-compiler-activation-feedstock
12. Cross compile all dependencies of conda-build and micromamba
13. Build a miniforge installer

@isuruf Is there more detailed information or development documentation available for our reference to make our work smoother? We are very sincere in promoting this project. Even though anaconda has not yet agreed to our application, we hope that miniforge will be running on my PC

[bismog]

So, I find now we can download from here: https://conda.loongnix.cn/miniforge, though I don't know the difference between V1(Miniforge-py38-Linux-loongarch64.sh) and V2(Miniforge3-24.1.2-Linux-loongarch64.sh), seems V1 with python3.8 and V2 with python3.12+.

[hmaarrfk]

So, I find now we can download from here: https://conda.loongnix.cn/miniforge, though I don't know the difference between V1(Miniforge-py38-Linux-loongarch64.sh) and V2(Miniforge3-24.1.2-Linux-loongarch64.sh), seems V1 with python3.8 and V2 with python3.12+.

These are not conda-forge projects.

Please contact the authors of the website you found directly for support.