sprint production release v2021.11.22

[harshad16]

Hello, Thoth-station!

This Issue would be used for the current sprint cycle production release.
By the end of the sprint cycle, we will consolidate the information of thoth-station components features upgrade and fixes in this issue.

[harshad16]

/kind documentation
/area release-eng
/triage accepted
/milestone 2021.11.22
/priority important-soon

[fridex]

Group step prescription

A new prescription of type "group step" allows prescribing rules in the resolution process based on a group of dependencies that are occurring in the resolution process. This is a higher-level abstraction that allows easier write prescriptions in cases when the resolution process should act differently based on a group of resolved dependencies (not necessarily a sub-graph) and the resolution order is not relevant. See the linked pull-requests to prescriptions repository for examples and online docs for more info.

• Implement group step prescription adviser#2156
• TensorFlow in versions >=2.6.0,<2.7.0 overpinned Keras dependency prescriptions#18544
• Group step pillow830 typeerror prescriptions#18545

[fridex]

Static source code analysis now aggregates also builtins

Starting this release, the recommendation engine warns users about the use of possibly dangerous parts of the Python programming language, such as exec and eval built-in functions that can introduce a security risk in the codebase.

• Aggregate also standard imports from user's code thamos#940
• Detect builtins used invectio#109
• Produce warnings when exec and eval are used prescriptions#17211

[fridex]

Container images endpoint and thamos images

WARNING: breaking change

The old endpoint /s2i/python on user-api has been deprecated and is no longer available. As an alternative we provide /container-images endpoint that lists available and analyzed Thoth container images users can use. The old endpoint was too specific for s2i (Source-To-Image). As the Thoth team provides multiple container images (s2i, predictable stacks, jupyter notebook container images, ...) the new endpoint is more generic and can be reused for different container image types.

As a subsequent change, Thamos CLI no longer supports thamos s2i command and users can use thamos images instead. As this is a breaking change, we encourage users to upgrade Thamos.

• Drop /s2i/python endpoint and promote container images endpoint user-api#1506
• Provide container images sub-command thamos#944

[fridex]

Dropped hardware endpoints on User API and dropped thamos hw command

WARNING: breaking change

We have dropped /hardware endpoint. It was no longer useful - recommendations can be still specific to hardware, but we do not directly expose information about the hardware on user-api endpoints. Hardware information can be encoded in the system on multiple places, to keep the system generic enough, we decided to drop the hardware listing endpoint. Also, the client tooling using this endpoint - thamos hw is no longer available. We encourage users to upgrade Thamos.

• Drop hardware endpoint user-api#1504
• Drop hardware subcommand thamos#943

[fridex]

Prescriptions now provide the ability to store metadata

Starting this release, prescriptions can state metadata. These metadata can be used by tools that integrate with prescriptions (generating prescriptions, bookkeeping state information, and such). The metadata field is not used during the actual resolution process. The feature was requested by @Gkrumbach07.

• Add metadata field to justification schema adviser#2166
• Introduce metadata in prescriptions adviser#2167

[pacospace]

jupyterlab-requirements v0.13.0

In this new release, users can see all justifications on the recommended stack directly in their notebook using %horus magic commands.
There are new commands to handle the kernels: check which ones are available, delete specific one or check what packages are available in each kernel.

• Release of version 0.13.0 jupyterlab-requirements#573

[pacospace]

new thamos whatprovides and discover command

Two new commands have been created thanks to the new endpoint added by @tlegen-k on User-API.

thamos whatprovides command -> allow users to identify the package from import names
thamos discover command -> allow users to identify what packages are required to run a certain script or series of scripts.

• Feature/use new endpoint thamos#945

[fridex]

Suggesting to use container images without known vulnerabilities

New logic in prescriptions-refresh-job computes which tags of container images available and produced by Thoth team do not have known vulnerabilities and suggests downgrading or upgrading the container image tag to users. This way users know which containerized environments are vulnerability-free.

• Provide suggestions to users based on statistics aggregated from requests done to the recommender prescriptions-refresh-job#41
• Compute alternatives to container images with vulnerabilities prescriptions-refresh-job#48

[mayaCostantini]

Making Thoth Datasets accessible via a public bucket and build a Jupyter Book gathering related notebooks and documentation

Thoth datasets were removed from the datasets repository and are now available on an Operate First Ceph public bucket accessible to users via the Jupyter Notebooks dedicated to their analysis. The repository has been modified to be able to build a Jupyter Book from the available notebooks and documentation, providing users with an organized and accessible way to explore the datasets.

• Add datasets image to Operate First datasets#31
• Make the project Meteor-compatible datasets#51

[harshad16]

we have completed the release of 2021.11.22 🎉 🎊 🥳

Features

Group step prescription

A new prescription of type "group step" allows prescribing rules in the resolution process based on a group of dependencies that are occurring in the resolution process. This is a higher-level abstraction that allows easier write prescriptions in cases when the resolution process should act differently based on a group of resolved dependencies (not necessarily a sub-graph) and the resolution order is not relevant. See the linked pull-requests to prescriptions repository for examples and online docs for more info.

• Implement group step prescription adviser#2156
• TensorFlow in versions >=2.6.0,<2.7.0 overpinned Keras dependency prescriptions#18544
• Group step pillow830 typeerror prescriptions#18545

Static source code analysis now aggregates also builtins

Starting this release, the recommendation engine warns users about the use of possibly dangerous parts of the Python programming language, such as exec and eval built-in functions that can introduce a security risk in the codebase.

• Aggregate also standard imports from user's code thamos#940
• Detect builtins used invectio#109
• Produce warnings when exec and eval are used prescriptions#17211

Container images endpoint and thamos images

WARNING: breaking change

The old endpoint /s2i/python on user-api has been deprecated and is no longer available. As an alternative we provide /container-images endpoint that lists available and analyzed Thoth container images users can use. The old endpoint was too specific for s2i (Source-To-Image). As the Thoth team provides multiple container images (s2i, predictable stacks, jupyter notebook container images, ...) the new endpoint is more generic and can be reused for different container image types.

As a subsequent change, Thamos CLI no longer supports thamos s2i command and users can use thamos images instead. As this is a breaking change, we encourage users to upgrade Thamos.

• Drop /s2i/python endpoint and promote container images endpoint user-api#1506
• Provide container images sub-command thamos#944

Dropped hardware endpoints on User API and dropped thamos hw command

WARNING: breaking change

We have dropped /hardware endpoint. It was no longer useful - recommendations can be still specific to hardware, but we do not directly expose information about the hardware on user-api endpoints. Hardware information can be encoded in the system on multiple places, to keep the system generic enough, we decided to drop the hardware listing endpoint. Also, the client tooling using this endpoint - thamos hw is no longer available. We encourage users to upgrade Thamos.

• Drop hardware endpoint user-api#1504
• Drop hardware subcommand thamos#943

Prescriptions now provide the ability to store metadata

Starting this release, prescriptions can state metadata. These metadata can be used by tools that integrate with prescriptions (generating prescriptions, bookkeeping state information, and such). The metadata field is not used during the actual resolution process. The feature was requested by @Gkrumbach07.

• Add metadata field to justification schema adviser#2166
• Introduce metadata in prescriptions adviser#2167

jupyterlab-requirements v0.13.0

In this new release, users can see all justifications on the recommended stack directly in their notebook using %horus magic commands.
There are new commands to handle the kernels: check which ones are available, delete specific one or check what packages are available in each kernel.

• Release of version 0.13.0 jupyterlab-requirements#573

new thamos whatprovides and discover command

Two new commands have been created thanks to the new endpoint added by @tlegen-k on User-API.

thamos whatprovides command -> allow users to identify the package from import names
thamos discover command -> allow users to identify what packages are required to run a certain script or series of scripts.

• Feature/use new endpoint thamos#945

Suggesting to use container images without known vulnerabilities

New logic in prescriptions-refresh-job computes which tags of container images available and produced by Thoth team do not have known vulnerabilities and suggests downgrading or upgrading the container image tag to users. This way users know which containerized environments are vulnerability-free.

• Provide suggestions to users based on statistics aggregated from requests done to the recommender prescriptions-refresh-job#41
• Compute alternatives to container images with vulnerabilities prescriptions-refresh-job#48

Making Thoth Datasets accessible via a public bucket and build a Jupyter Book gathering related notebooks and documentation

Thoth datasets were removed from the datasets repository and are now available on an Operate First Ceph public bucket accessible to users via the Jupyter Notebooks dedicated to their analysis. The repository has been modified to be able to build a Jupyter Book from the available notebooks and documentation, providing users with an organized and accessible way to explore the datasets.

• Add datasets image to Operate First datasets#31
• Make the project Meteor-compatible datasets#51

Component Updates

• user-api: v0.28.1

  • Drop hardware endpoint user-api#1504
  • updating to https routes for pre-commit user-api#1507
  • Drop /s2i/python endpoint and promote container images endpoint user-api#1506
  • Adjust output type for /python/imports endpoint user-api#1508

• kebechet: v0.16.6

  • Track advise status using GitHub issues kebechet#893
  • patch the fasttext version for build fix kebechet#900
  • if we have already cloned to dir, return repo from dir kebechet#908
  • add kebechet- to all branch names kebechet#916
  • No computed changelog kebechet#919

• prescriptions-refresh-job: v0.4.0

  • Compute alternatives to container images with vulnerabilities prescriptions-refresh-job#48
  • Keep datetime in the Quay security timestamp boot in metadata prescriptions-refresh-job#49
  • Sort response from Quay API to keep prescription changes consistent prescriptions-refresh-job#50

Thanks for the amazing work everyone. 💯