Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Starts work on adding recipe upgrade feature #539

Conversation

schuylermartin45
Copy link
Contributor

@schuylermartin45 schuylermartin45 commented May 16, 2024

Description

First-pass implementation supporting the V1 recipe standard (used by rattler-build, as discussed in #486

As of writing this PR (currently in the DRAFT stage), there are a few issues:
1. conda-recipe-manager is not currently in the conda-forge channel. There is an open PR waiting for a review to add it in.
2. Locally, I can't get ruamel to dump souschef's version of the recipe data into a StringIO stream. I had many issues getting conda to use my local copy of conda-recipe-manager and I think it's related to that. I have ruamel.yaml.jinja installed in the local environment, but I still get an error. Here are some logs for more details:

(grayskull) smartin@Schuylers-MacBook-Pro:grayskull> grayskull pypi -u types-toml                       



#### Initializing recipe for types-toml (pypi) ####

Recovering metadata from pypi...
Starting the download of the sdist package types-toml
types-toml 100% Time:  0:00:00   1.8 MiB/s|####################################################################################|
Checking for pyproject.toml
pyproject.toml not found.
Recovering information from setup.py
Executing injected distutils...
Recovering metadata from setup.cfg
No data was recovered from setup.py. Forcing to execute the setup.py as script
Recovering metadata from setup.cfg
Checking >> -- 100% |#                                                                                  |[Elapsed Time: 0:00:00]
Recovering license info from spdx.org ...
INFO:Best match for license Apache-2.0 was Apache-2.0.
Best matches: [('Apache-2.0', 100.0, 0), ('Apache-1.0', 90.0, 1), ('Apache-1.1', 80.0, 2), ('Apache License 1.0', 67.5, 4), ('Zend-2.0', 55.55555555555556, 3)]
INFO:Github url: https://api.github.com/repos/python/typeshed/license?ref=0.10.8.20240310 - recovering license info
Recovering license information from github...
Recovering license info from repository...
Cloning into '/var/folders/mj/y6v4_y555yx3_370k4sk0_bh0000gp/T/gs-clone-repo-amc55x71'...
fatal: Remote branch 0.10.8.20240310 not found in upstream origin
Recovering license info from repository...
Cloning into '/var/folders/mj/y6v4_y555yx3_370k4sk0_bh0000gp/T/gs-clone-repo-a_k02yuu'...
fatal: Remote branch v0.10.8.20240310 not found in upstream origin
License type: Apache-2.0
License file: []
Build requirements:
  <none>
Host requirements:
  - python >=3.8
  - pip
Run requirements:
  - python >=3.8

RED: Package names not available on conda-forge
YELLOW: PEP-725 PURLs that did not map to known package
GREEN: Packages available on conda-forge

Maintainers:
   - schuylermartin45
Traceback (most recent call last):
  File "/Users/smartin/miniconda3/envs/grayskull/bin/grayskull", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/smartin/work/grayskull/grayskull/main.py", line 295, in main
    generate_recipes_from_list(args.pypi_packages, args)
  File "/Users/smartin/work/grayskull/grayskull/main.py", line 340, in generate_recipes_from_list
    generate_recipe(recipe, config, args.output, args.use_v1_format)
  File "/Users/smartin/work/grayskull/grayskull/utils.py", line 230, in generate_recipe
    upgrade_v0_recipe_to_v1(recipe, recipe_path)
  File "/Users/smartin/work/grayskull/grayskull/utils.py", line 247, in upgrade_v0_recipe_to_v1
    yaml.dump(recipe.yaml, recipe_stream)
  File "/Users/smartin/miniconda3/envs/grayskull/lib/python3.11/site-packages/ruamel/yaml/main.py", line 574, in dump
    return self.dump_all([data], stream, transform=transform)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/smartin/miniconda3/envs/grayskull/lib/python3.11/site-packages/ruamel/yaml/main.py", line 583, in dump_all
    self._context_manager.dump(data)
  File "/Users/smartin/miniconda3/envs/grayskull/lib/python3.11/site-packages/ruamel/yaml/main.py", line 913, in dump
    self.init_output(data)
  File "/Users/smartin/miniconda3/envs/grayskull/lib/python3.11/site-packages/ruamel/yaml/main.py", line 906, in init_output
    self._yaml.get_serializer_representer_emitter(self._output, tlca)
  File "/Users/smartin/miniconda3/envs/grayskull/lib/python3.11/site-packages/ruamel/yaml/main.py", line 644, in get_serializer_representer_emitter
    self.emitter.stream = stream
    ^^^^^^^^^^^^^^^^^^^
  File "/Users/smartin/miniconda3/envs/grayskull/lib/python3.11/site-packages/ruamel/yaml/jinja2/__plug_in__.py", line 88, in stream
    self, Rewriter(val, getattr(self.dumper, '_plug_in_' + typ))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'YAML' object has no attribute '_plug_in_jinja2'

The package that this commit relies on is not currently available in
conda-forge. Viewer discretion is advised.

- First-pass attempt at integrating `conda-recipe-manager` into Grayskull for
  the purposes of generating recipes in the V1 format.
@schuylermartin45
Copy link
Contributor Author

Update: I rebuilt my local environment this morning now that conda-recipe-manager is in conda-forge.

Unfortunately I am still getting:

AttributeError: 'YAML' object has no attribute '_plug_in_jinja2'

when I try to run the conversion flag. Does anyone know if I have a missing dependency locally? Do I have to install a plugin not listed in the environment.yaml file for ruamel?

@schuylermartin45
Copy link
Contributor Author

Update: I rebuilt my local environment this morning now that conda-recipe-manager is in conda-forge.

Unfortunately I am still getting:

AttributeError: 'YAML' object has no attribute '_plug_in_jinja2'

when I try to run the conversion flag. Does anyone know if I have a missing dependency locally? Do I have to install a plugin not listed in the environment.yaml file for ruamel?

Here are the ruamel packages I have installed in this environment:

ruamel.yaml               0.18.6          py312he37b823_0    conda-forge
ruamel.yaml.clib          0.2.8           py312he37b823_0    conda-forge
ruamel.yaml.jinja2        0.2.4                      py_1    conda-forge

@schuylermartin45
Copy link
Contributor Author

@marcelotrevisani I'd hate to bother you directly, but have you seen these issues before? I get the feeling that I'm missing some dependencies for ruamel and haven't found much when searching with this error message.

Here's how I've configured my development:

conda env create -f environment.yaml --name grayskull --yes
conda run --name grayskull pip install -e .
conda activate grayskull

I've also tried forcing the environment to use 3.11 by editing environment.yaml:

  - python >=3.8,<3.12

@marcelotrevisani
Copy link
Member

Hi, sorry, I have been busy in my personal life and job.
I didn't have time to take a look, but did you try to downngrade the version of your ruamel?

@schuylermartin45
Copy link
Contributor Author

Hi, sorry, I have been busy in my personal life and job. I didn't have time to take a look, but did you try to downngrade the version of your ruamel?

No I have not and did not think to do that. Do you know off the top of your head what the last known good version was? I don't see it listed in the environment file so I'm guessing it transitively is included with sous-chef?

@schuylermartin45
Copy link
Contributor Author

Hi, sorry, I have been busy in my personal life and job. I didn't have time to take a look, but did you try to downngrade the version of your ruamel?

No I have not and did not think to do that. Do you know off the top of your head what the last known good version was? I don't see it listed in the environment file so I'm guessing it transitively is included with sous-chef?

Yeah I see it in sous-chef as >=0.15.3 so I'll start from there and see what happens. Thanks!

@schuylermartin45
Copy link
Contributor Author

schuylermartin45 commented Jun 5, 2024

Latest update: I've struggled to get conda to use an older version of ruamel.yaml so I have started to use venv and pip to experiment. The oldest version of ruamel.yaml I've used so far is 0.16.10 in this setup and I still get the same error message. pip would not allow me to install 0.15.3 so I'm going to try to see what the oldest version between the two is that pip will let me revert to.

I also ran a brief experiment yesterday and tried to have ruamel dump to a file and not a string stream. That causes the same error, but locally souschef's library is performing the same logic on my local environment without issue. I get the impression that somehow, when called from grayskull, ruamel can't find the jinja2 plugin that is readily accessible in souschef.

@schuylermartin45
Copy link
Contributor Author

pip will not let me install ruamel@0.15.* but will let me try ruamel@0.16.* and 0.16.1 returns the same error. I think I've ruled out a version incompatibility issue at this point.

@dhirschfeld
Copy link

dhirschfeld commented Jun 5, 2024

IIRC if you conda install a local .bz2/.conda file it will just do it, ignoring all constraints.

@schuylermartin45
Copy link
Contributor Author

After talking with @beeankha, we found a workaround. I'm not super proud of it, but I've gotten around the JINJA issue by reading the recipe file produced by souschef::recipe.save(), converting that string, and then rewriting the file with the converted text.

Like I said, it's a hack but it works for the time being. Arguably we could modify this further by having the upgrade option return both a meta.yaml and a recipe.yaml file as we transition between formats.

I'll be moving the PR out of the Draft state now that we have a working solution.

@schuylermartin45 schuylermartin45 marked this pull request as ready for review June 18, 2024 19:22
@schuylermartin45 schuylermartin45 requested a review from a team as a code owner June 18, 2024 19:22
@marcelotrevisani marcelotrevisani merged commit 3b83710 into conda:main Jul 28, 2024
5 of 9 checks passed
@schuylermartin45 schuylermartin45 deleted the smartin_adding_crm_conversion_support branch July 29, 2024 15:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants