Adding pyflamegpu #24483

Robadob · 2023-11-13T16:16:44Z

Checklist

…img)

conda-forge-webservices · 2023-11-13T16:17:06Z

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipes/pyflamegpu, recipes/pyflamegpu-no_seatbelts, recipes/pyflamegpu-visualisation, recipes/pyflamegpu-visualisation-no_seatbelts) and found some lint.

Here's what I've got...

For recipes/pyflamegpu:

The top level meta keys are in an unexpected order. Expecting ['package', 'source', 'build', 'requirements', 'test', 'about', 'extra'].
Selectors are suggested to take a <two spaces>#<one space>[<expression>] form. See lines [17, 18, 19, 44, 45, 46, 47, 48, 50, 51, 52, 53]
The recipe must have a build/number section.
When defining a source/url please add a sha256, sha1 or md5 checksum (sha256 preferably).
There are too few lines. There should be one empty line at the end of the file.

For recipes/pyflamegpu:

The pypa build package has been renamed to python-build.

For recipes/pyflamegpu-no_seatbelts:

The top level meta keys are in an unexpected order. Expecting ['package', 'source', 'build', 'requirements', 'test', 'about', 'extra'].
Selectors are suggested to take a <two spaces>#<one space>[<expression>] form. See lines [17, 18, 19, 47, 48, 49, 50, 51, 53, 54, 55, 56]
The recipe must have a build/number section.
When defining a source/url please add a sha256, sha1 or md5 checksum (sha256 preferably).
There are too few lines. There should be one empty line at the end of the file.

For recipes/pyflamegpu-no_seatbelts:

The pypa build package has been renamed to python-build.

For recipes/pyflamegpu-visualisation:

The top level meta keys are in an unexpected order. Expecting ['package', 'source', 'build', 'requirements', 'test', 'about', 'extra'].
Selectors are suggested to take a <two spaces>#<one space>[<expression>] form. See lines [18, 19, 20, 21, 22, 39, 56, 57, 58, 59, 60, 62, 63, 64, 65]
The recipe must have a build/number section.
When defining a source/url please add a sha256, sha1 or md5 checksum (sha256 preferably).
There are too few lines. There should be one empty line at the end of the file.

For recipes/pyflamegpu-visualisation:

The pypa build package has been renamed to python-build.

For recipes/pyflamegpu-visualisation-no_seatbelts:

The top level meta keys are in an unexpected order. Expecting ['package', 'source', 'build', 'requirements', 'test', 'about', 'extra'].
Selectors are suggested to take a <two spaces>#<one space>[<expression>] form. See lines [18, 19, 20, 21, 22, 39, 57, 58, 59, 60, 61, 63, 64, 65, 66]
The recipe must have a build/number section.
When defining a source/url please add a sha256, sha1 or md5 checksum (sha256 preferably).
There are too few lines. There should be one empty line at the end of the file.

For recipes/pyflamegpu-visualisation-no_seatbelts:

The pypa build package has been renamed to python-build.

Not yet addressed sha, as development

conda-forge-webservices · 2023-11-13T16:35:40Z

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipes/pyflamegpu, recipes/pyflamegpu-no_seatbelts, recipes/pyflamegpu-visualisation, recipes/pyflamegpu-visualisation-no_seatbelts) and found some lint.

Here's what I've got...

For recipes/pyflamegpu:

When defining a source/url please add a sha256, sha1 or md5 checksum (sha256 preferably).

For recipes/pyflamegpu:

The pypa build package has been renamed to python-build.

For recipes/pyflamegpu-no_seatbelts:

When defining a source/url please add a sha256, sha1 or md5 checksum (sha256 preferably).

For recipes/pyflamegpu-no_seatbelts:

The pypa build package has been renamed to python-build.

For recipes/pyflamegpu-visualisation:

When defining a source/url please add a sha256, sha1 or md5 checksum (sha256 preferably).

For recipes/pyflamegpu-visualisation:

The pypa build package has been renamed to python-build.

For recipes/pyflamegpu-visualisation-no_seatbelts:

When defining a source/url please add a sha256, sha1 or md5 checksum (sha256 preferably).

For recipes/pyflamegpu-visualisation-no_seatbelts:

The pypa build package has been renamed to python-build.

Robadob · 2023-11-13T16:36:46Z

@conda-forge-admin, please ping conda-forge/staged-recipes

Questions primarily relate to CUDA and build variants/metapackages, this is a Python/C++ hybrid package that uses CUDA.

I've been working through creating recipes, and have run into a few questions (although some are more general). I don't believe any of these are answered in the knowledge base

Our package has four build variants (2x2), each independently compiled from source, I've named them in what I believe to be a meta-package style but would like clarity on whether this is the correct process. I can't find any particularly clear documentation on formatting build variants.
Our package depends on CUDA, to get the local build to succeed I have currently been using the linux64 config. Using linux64_cuda112 prevents us from building with CUDA 12.0. Ideally we would like to have both CUDA 11.2 and CUDA 12.0 build variants for compatibility reasons, however CUDA 11.2 requires the cudatoolkit package and CUDA 12.0 has split CUDA packages. It's not clear how we can detect this to filter packages within meta.yml.
The above linked "creating recipes" states:

In case your project has tests included, you need to decide if these tests should be executed while building the conda-forge feedstock.

Our package relies on CUDA, and hence tests specified in meta.yml won't be able to execute on CI runners. It's not clear how the execution of tests on conda-forge should be disabled.

Our Windows builds are currently failing to detect CUDA support at CMake configure time on conda-forge CI. I tried alot of configurations before I found a combination that works with local vanilla conda build, and don't immediately recognise what would be the problem on your CI (I have added cuda-version >=12.0 to meta.yml since then). Knowledge base references __cuda, but I think that's related to run constraints, so shouldn't interfere with build.

conda-forge-webservices · 2023-11-13T16:36:49Z

Hi! This is the friendly automated conda-forge-webservice.

I was asked to ping @conda-forge/staged-recipes and so here I am doing that.

Before this change, tested that package fully builds on Linux with tests commented out.

conda-forge-webservices · 2023-11-14T08:47:23Z

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipes/pyflamegpu, recipes/pyflamegpu-no_seatbelts, recipes/pyflamegpu-visualisation, recipes/pyflamegpu-visualisation-no_seatbelts) and found some lint.

Here's what I've got...

For recipes/pyflamegpu:

When defining a source/url please add a sha256, sha1 or md5 checksum (sha256 preferably).

For recipes/pyflamegpu-no_seatbelts:

When defining a source/url please add a sha256, sha1 or md5 checksum (sha256 preferably).

For recipes/pyflamegpu-visualisation:

When defining a source/url please add a sha256, sha1 or md5 checksum (sha256 preferably).

For recipes/pyflamegpu-visualisation-no_seatbelts:

When defining a source/url please add a sha256, sha1 or md5 checksum (sha256 preferably).

conda-forge-webservices · 2023-11-14T09:24:53Z

Hi! This is the friendly automated conda-forge-webservice.

I was asked to ping @conda-forge/staged-recipes and so here I am doing that.

conda-forge-webservices · 2023-11-14T09:25:43Z

Hi! This is the friendly automated conda-forge-webservice.

I was asked to ping @conda-forge/staged-recipes and so here I am doing that.

mondus · 2023-11-17T11:34:20Z

I agree to be a maintainer.

carterbox · 2024-04-04T18:58:18Z

Our package has four build variants (2x2), each independently compiled from source, I've named them in what I believe to be a meta-package style but would like clarity on whether this is the correct process. I can't find any particularly clear documentation on formatting build variants.

You can optionally add keywords to the build string of each build variant in order to make it easier to see which variant is installed in an environment. i.e. pytorch-2.1.2-cuda120_py39h832322d_303.conda. However, this is not strictly necessary because each variant will have a unique build string which is generated from hashing the dependencies used to build the package.

Our package depends on CUDA, to get the local build to succeed I have currently been using the linux64 config. Using linux64_cuda112 prevents us from building with CUDA 12.0. Ideally we would like to have both CUDA 11.2 and CUDA 12.0 build variants for compatibility reasons, however CUDA 11.2 requires the cudatoolkit package and CUDA 12.0 has split CUDA packages. It's not clear how we can detect this to filter packages within meta.yml.

Our infrastructure creates separate build jobs for each CUDA version. We have a tool called conda-smithy which parses recipes and decided which build variants to automatically insert into the build matrix. The same process is used for building for multiple python versions.

The above linked "creating recipes" states:

In case your project has tests included, you need to decide if these tests should be executed while building the conda-forge feedstock.

Our package relies on CUDA, and hence tests specified in meta.yml won't be able to execute on CI runners. It's not clear how the execution of tests on conda-forge should be disabled.

We do not offer build runners with GPUs, so you need to skip any tests that require loading the CUDA driver.

Our Windows builds are currently failing to detect CUDA support at CMake configure time on conda-forge CI. I tried alot of configurations before I found a combination that works with local vanilla conda build, and don't immediately recognise what would be the problem on your CI (I have added cuda-version >=12.0 to meta.yml since then). Knowledge base references __cuda, but I think that's related to run constraints, so shouldn't interfere with build.

The build runners do not have any GPUs. Ensure that you are using our CMAKE_ARGS environment variable and the compilers that are set to the environment variables CC, CXX, etc. Also, only the LINUX runner is setup for CUDA here in staged recipes. There is no runner for Windows + CUDA until your recipe is given it's own feedstock.

carterbox · 2024-04-04T19:13:43Z

Here in staged recipes, the package will only be built against CUDA 11.2 on linux. Once in the feedstock, it will be rerendered and then it will be built against 11.2, 11.8, and 12.x.

Here are some examples that are similar to your recipes:

#23415 -> https://github.com/conda-forge/carterbox-torch-radon-feedstock

https://github.com/conda-forge/libmagma-feedstock

carterbox · 2024-04-04T19:22:50Z

recipes/pyflamegpu/conda_build_config.yaml

+# https://docs.conda.io/projects/conda-build/en/3.21.x/resources/compiler-tools.html#using-your-customized-compiler-package-with-conda-build-3
+# https://github.com/rapidsai/cuml/blob/branch-23.12/conda/recipes/libcuml/conda_build_config.yaml
+# https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/main/recipe/conda_build_config.yaml
+cuda_compiler:
+  - cuda-nvcc # requires 'conda build -c nvidia conda' on Windows
+  #- nvcc # requires 'conda build -c nvidia conda' on Windows
+cuda_compiler_version:
+  #- 11.2                       # [(linux or win64) and os.environ.get("CF_CUDA_ENABLED", "False") == "True"]
+  - 12.0                       # [(linux or win64) and os.environ.get("CF_CUDA_ENABLED", "False") == "True"]
+  - 12.0                       
+cuda_compiler_version_min:
+  #- 11.2                       # [linux or win64]
+  - 12.0                       # [linux or win64]
+  - 12.0                       
+
+c_compiler: # [win]
+  - vs2022 # [win]
+cxx_compiler: # [win]
+  - vs2022 # [win]


Delete this file. CUDA and compiler variants are managed by our infrastructure.

carterbox · 2024-04-04T19:23:51Z

recipes/pyflamegpu/meta.yaml

+  missing_dso_whitelist:
+    - $RPATH/libcuda.so.1  # [linux] Ignore as this comes from CUDA driver
+    - $RPATH/libnvrtc.so.12  # [linux] Conda-build can't find this in run reqs cuda-cudart
+    - $RPATH/libcudart.so.12  # [linux] Conda-build can't find this in run reqs cuda-nvrtc
+    - $RPATH/nvrtc64_120_0.dll  # [win64] Conda-build can't find this in run reqs cuda-cudart
+    - $RPATH/cudart64_12.dll  # [win64] Conda-build can't find this in run reqs cuda-nvrtc


Suggested change

missing_dso_whitelist:

- $RPATH/libcuda.so.1 # [linux] Ignore as this comes from CUDA driver

- $RPATH/libnvrtc.so.12 # [linux] Conda-build can't find this in run reqs cuda-cudart

- $RPATH/libcudart.so.12 # [linux] Conda-build can't find this in run reqs cuda-nvrtc

- $RPATH/nvrtc64_120_0.dll # [win64] Conda-build can't find this in run reqs cuda-cudart

- $RPATH/cudart64_12.dll # [win64] Conda-build can't find this in run reqs cuda-nvrtc

These should not be needed because in staged-recipes there is only one CUDA variant available: CUDA 11.2.

carterbox · 2024-04-04T19:24:46Z

recipes/pyflamegpu/meta.yaml

+build:
+  number: 0
+  script_env:


Suggested change

build:

number: 0

script_env:

build:

number: 0

skip: true # [cuda_compiler_version == "None"]

skip: true # [py < 310]

script_env:

Skip building the default non-cuda variant forever. Skip lower python versions for faster debugging for now, then unskip later.

carterbox · 2024-04-04T19:26:41Z

recipes/pyflamegpu/meta.yaml

+  ignore_run_exports:
+    - python  # [linux] Not clear why conda thinks this isn't used
+    - astpretty  # [linux, win64] Not clear why conda thinks this isn't used
+    - vc14_runtime  # [win64] Not clear why conda thinks this isn't used (we aren't static linking)
+    - ucrt  # [win64] Not clear why conda thinks this isn't used (we aren't static linking)


Suggested change

ignore_run_exports:

- python # [linux] Not clear why conda thinks this isn't used

- astpretty # [linux, win64] Not clear why conda thinks this isn't used

- vc14_runtime # [win64] Not clear why conda thinks this isn't used (we aren't static linking)

- ucrt # [win64] Not clear why conda thinks this isn't used (we aren't static linking)

This is not the purpose of ignore_run_exports. You should only add host packages here that you know are NOT being used at runtime. e.g. You are using a C library that has some header-only APIs.

carterbox · 2024-04-04T19:28:49Z

recipes/pyflamegpu/meta.yaml

+    - cuda-cudart-dev  # [linux]
+    - cuda-nvrtc-dev  # [linux]
+    - libcurand-dev  # [linux]


Suggested change

- cuda-cudart-dev # [linux]

- cuda-nvrtc-dev # [linux]

- libcurand-dev # [linux]

{% if cuda_major == 12 %}

- cuda-cudart-dev # [linux]

- cuda-nvrtc-dev # [linux]

- libcurand-dev # [linux]

{% endif %}

These packages are only for CUDA >= 12. Our package structure changed for CUDA 12. Before that, everything was shipped as one monolithic cudatoolkit package.

Also, these are libraries, so they belong in the HOST requirements section.

carterbox · 2024-04-04T19:29:30Z

recipes/pyflamegpu/meta.yaml

+    - wheel
+    - python-build
+  run:
+    - cuda-version >=12.0


Suggested change

- cuda-version >=12.0

The cuda version is run_exported by our cuda compiler package.

carterbox · 2024-04-04T19:33:32Z

recipes/pyflamegpu/meta.yaml

+  license_family: MIT
+  license_file: LICENSE.md
+  doc_url: https://docs.flamegpu.com/
+  dev_url: https://github.com/FLAMEGPU/FLAMEGPU2


Looks like there are separate CXX and Python APIs? Can you build the CXX libraries first, then link the Python modules to them dynamically?

Robadob and others added 4 commits November 13, 2023 11:38

draft recipes for conda-forge.

fdb79a9

conda forge docker yml

548f2fb

syntax err

029bed0

This now builds first package but fails to test (as not in CUDA base …

58408b3

…img)

lint fixes

9bd7336

Not yet addressed sha, as development

github-actions bot added review-requested staged-recipes labels Nov 13, 2023

Robadob mentioned this pull request Nov 13, 2023

conda packaging (experimental WIP) FLAMEGPU/FLAMEGPU2#1131

Draft

4 tasks

Missed build to python-build in prev lint pass

624b95a

Before this change, tested that package fully builds on Linux with tests commented out.

carterbox added the python-c label Apr 4, 2024

carterbox requested changes Apr 4, 2024

View reviewed changes

carterbox removed the review-requested label Apr 4, 2024

github-actions bot added the Awaiting author contribution label Apr 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding pyflamegpu #24483

Adding pyflamegpu #24483

Robadob commented Nov 13, 2023 •

edited

Loading

conda-forge-webservices bot commented Nov 13, 2023

conda-forge-webservices bot commented Nov 13, 2023

Robadob commented Nov 13, 2023 •

edited

Loading

conda-forge-webservices bot commented Nov 13, 2023

conda-forge-webservices bot commented Nov 14, 2023

conda-forge-webservices bot commented Nov 14, 2023

conda-forge-webservices bot commented Nov 14, 2023

mondus commented Nov 17, 2023

carterbox commented Apr 4, 2024 •

edited

Loading

carterbox commented Apr 4, 2024 •

edited

Loading

carterbox Apr 4, 2024

carterbox Apr 4, 2024

carterbox Apr 4, 2024

carterbox Apr 4, 2024

carterbox Apr 4, 2024

carterbox Apr 4, 2024

carterbox Apr 4, 2024

Adding pyflamegpu #24483

Are you sure you want to change the base?

Adding pyflamegpu #24483

Conversation

Robadob commented Nov 13, 2023 • edited Loading

conda-forge-webservices bot commented Nov 13, 2023

conda-forge-webservices bot commented Nov 13, 2023

Robadob commented Nov 13, 2023 • edited Loading

conda-forge-webservices bot commented Nov 13, 2023

conda-forge-webservices bot commented Nov 14, 2023

conda-forge-webservices bot commented Nov 14, 2023

conda-forge-webservices bot commented Nov 14, 2023

mondus commented Nov 17, 2023

carterbox commented Apr 4, 2024 • edited Loading

carterbox commented Apr 4, 2024 • edited Loading

carterbox Apr 4, 2024

Choose a reason for hiding this comment

carterbox Apr 4, 2024

Choose a reason for hiding this comment

carterbox Apr 4, 2024

Choose a reason for hiding this comment

carterbox Apr 4, 2024

Choose a reason for hiding this comment

carterbox Apr 4, 2024

Choose a reason for hiding this comment

carterbox Apr 4, 2024

Choose a reason for hiding this comment

carterbox Apr 4, 2024

Choose a reason for hiding this comment

Robadob commented Nov 13, 2023 •

edited

Loading

Robadob commented Nov 13, 2023 •

edited

Loading

carterbox commented Apr 4, 2024 •

edited

Loading

carterbox commented Apr 4, 2024 •

edited

Loading