Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Software Versions Tools #141

Merged
merged 4 commits into from Mar 16, 2022
Merged

Conversation

albert17
Copy link
Contributor

@albert17 albert17 commented Mar 10, 2022

Running the tool

python versions.py -v 22.03

Will produce output

{'merlin-training': {'CUDA': 'V11.6.55', 'rmm': '21.12.0a0+31.g0acbd51', 'cudf': '21.12.0a0+293.g0930f712e6', 'merlin-core': '0.1.1+3.gee1d59d', 'nvtabular': '0.11.0', 'transformers4rec': 'N/A', 'models': 'N/A', 'hugectr': '3.2'}, 'merlin-tensorflow-training': {'CUDA': 'V11.6.55', 'rmm': '21.12.0a0+31.g0acbd51', 'cudf': '21.12.0a0+293.g0930f712e6', 'merlin-core': '0.1.1+3.gee1d59d', 'nvtabular': '0.11.0', 'transformers4rec': '0.1.6', 'models': '0.2.0', 'hugectr': 'N/A'}, 'merlin-pytorch-training': {'CUDA': 'V11.6.55', 'rmm': '21.12.0a0+31.g0acbd51', 'cudf': '21.12.0a0+293.g0930f712e6', 'merlin-core': '0.1.1+3.gee1d59d', 'nvtabular': '0.11.0', 'transformers4rec': '0.1.6', 'models': '0.2.0', 'hugectr': 'N/A'}, 'merlin-inference': {'CUDA': 'V11.6.55', 'rmm': '21.12.0', 'cudf': '21.12.2', 'merlin-core': '0.1.1+3.gee1d59d', 'nvtabular': '0.11.0', 'transformers4rec': '0.1.6', 'models': '0.2.0', 'hugectr': 'N/A'}, 'merlin-tensorflow-inference': {'CUDA': 'V11.6.55', 'rmm': '21.12.0', 'cudf': '21.12.2', 'merlin-core': '0.1.1+3.gee1d59d', 'nvtabular': '0.11.0', 'transformers4rec': '0.1.6', 'models': '0.2.0', 'hugectr': 'N/A'}, 'merlin-pytorch-inference': {'CUDA': 'V11.6.55', 'rmm': '21.12.0', 'cudf': '21.12.2', 'merlin-core': '0.1.1+3.gee1d59d', 'nvtabular': '0.11.0', 'transformers4rec': '0.1.6', 'models': '0.2.0', 'hugectr': 'N/A'}}

We can do whatever with this dictionary: Create Json, Markdown, ...

We can also add more stuff we need to this script

EDIT: Output format updated to markdown table

Container CUDA rmm cudf merlin-core nvtabular transformers4rec models hugectr
nvcr.io/nvidia/merlin/merlin-training:22.03 V11.6.55 21.12.0a0+31.g0acbd51 21.12.0a0+293.g0930f712e6 0.1.1+3.gee1d59d 0.11.0 N/A N/A 3.2
nvcr.io/nvidia/merlin/merlin-tensorflow-training:22.03 V11.6.55 21.12.0a0+31.g0acbd51 21.12.0a0+293.g0930f712e6 0.1.1+3.gee1d59d 0.11.0 0.1.6 0.2.0 N/A
nvcr.io/nvidia/merlin/merlin-pytorch-training:22.03 V11.6.55 21.12.0a0+31.g0acbd51 21.12.0a0+293.g0930f712e6 0.1.1+3.gee1d59d 0.11.0 0.1.6 0.2.0 N/A
nvcr.io/nvidia/merlin/merlin-inference:22.03 V11.6.55 21.12.0 21.12.2 0.1.1+3.gee1d59d 0.11.0 0.1.6 0.2.0 N/A
nvcr.io/nvidia/merlin/merlin-tensorflow-inference:22.03 V11.6.55 21.12.0 21.12.2 0.1.1+3.gee1d59d 0.11.0 0.1.6 0.2.0 N/A
nvcr.io/nvidia/merlin/merlin-pytorch-inference:22.03 V11.6.55 21.12.0 21.12.2 0.1.1+3.gee1d59d 0.11.0 0.1.6 0.2.0 N/A

@albert17 albert17 linked an issue Mar 10, 2022 that may be closed by this pull request
@albert17
Copy link
Contributor Author

@viswa-nvidia @mikemckiernan for feedback

@benfred @EvenOldridge @jperez999 to see if we can use this for something else or you would like to extend in a certain way

ci/versions.py Outdated
info[cont]["models"] = get_pythonpkg_version(container, "models")
# Get HugeCTR
info[cont]["hugectr"] = get_pythonpkg_version(container, "hugectr")
print(info)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm tying what might be my user error to this line. Is there a print or library that can use double-quotation marks instead of the apostrophe character?

jq . < /tmp/orig
parse error: Invalid numeric literal at line 1, column 19

jq . <(sed -e "s/'/\"/g" /tmp/orig)
{
  "merlin-training": {
    "CUDA": "V11.6.55",
    ...

I googled. Maybe json.dumps().

Huge thanks, btw.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the structures, to output a Markdown table. See comments

@albert17
Copy link
Contributor Author

albert17 commented Mar 10, 2022

@EvenOldridge @mikemckiernan @viswa-nvidia See updates. This is the Markdown output table dynamically generated by the script using existing containers. We can use it in the following ways

  1. Generate accurate release matrix. Based in which is actually installed in the container.
  2. Detect mismatches between versions we are setting and versions that are actually finally being installed
  3. Detect mismatches across containers

It is easy to extend adding/removing packaged. We can change formatting as you wish

Container CUDA rmm cudf merlin-core nvtabular transformers4rec models hugectr
nvcr.io/nvidia/merlin/merlin-training:22.03 V11.6.55 21.12.0a0+31.g0acbd51 21.12.0a0+293.g0930f712e6 0.1.1+3.gee1d59d 0.11.0 N/A N/A 3.2
nvcr.io/nvidia/merlin/merlin-tensorflow-training:22.03 V11.6.55 21.12.0a0+31.g0acbd51 21.12.0a0+293.g0930f712e6 0.1.1+3.gee1d59d 0.11.0 0.1.6 0.2.0 N/A
nvcr.io/nvidia/merlin/merlin-pytorch-training:22.03 V11.6.55 21.12.0a0+31.g0acbd51 21.12.0a0+293.g0930f712e6 0.1.1+3.gee1d59d 0.11.0 0.1.6 0.2.0 N/A
nvcr.io/nvidia/merlin/merlin-inference:22.03 V11.6.55 21.12.0 21.12.2 0.1.1+3.gee1d59d 0.11.0 0.1.6 0.2.0 N/A
nvcr.io/nvidia/merlin/merlin-tensorflow-inference:22.03 V11.6.55 21.12.0 21.12.2 0.1.1+3.gee1d59d 0.11.0 0.1.6 0.2.0 N/A
nvcr.io/nvidia/merlin/merlin-pytorch-inference:22.03 V11.6.55 21.12.0 21.12.2 0.1.1+3.gee1d59d 0.11.0 0.1.6 0.2.0 N/A

@albert17
Copy link
Contributor Author

@mikemckiernan @EvenOldridge Discussed with Even I am gonna try to expand this to creating an automatic PR about the release

@albert17
Copy link
Contributor Author

@mikemckiernan @EvenOldridge I updated the script. Now creates a PR with the table. Look at: #147

@albert17 albert17 merged commit 4d44c7c into NVIDIA-Merlin:main Mar 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Container software version
3 participants