Skip to content
Package and package stack adviser for the Thoth project
Python Shell
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github added the standard github configuration and a CODEOWNERS file Jul 17, 2019
docs/source Extend docs with MDP Jan 13, 2020
openshift Merge branch 'master' into rename-template Jan 20, 2020
tests Introduce version constrain sieve pipeline unit Jan 16, 2020
thoth/adviser Try its best to always come up with a latest stack Jan 20, 2020
.coafile Add more packages to cache config file Sep 17, 2019
.gitignore 🙈 ignoring the xml report coverage file Dec 17, 2019
.thoth.yaml Use RHEL instead of UBI Dec 6, 2019
.thothTemplate.yaml Update Thoth configuration file and Thoth's s2i configuration Dec 6, 2019
.zuul.yaml Add build trigger using generic webhook Jun 17, 2019
CHANGELOG.md Release of version 0.7.3 Jan 14, 2020
LICENSE Add LICENSE and license headers Apr 24, 2018
MANIFEST.in Do not forget to package requirements.txt Dec 3, 2018
OWNERS Create OWNERS May 17, 2018
Pipfile Introduce version constrain sieve pipeline unit Jan 16, 2020
Pipfile.lock
README.rst Add ability to block pipeline units during pipeline build Jan 10, 2020
app.sh Limit number of software stacks to 1 on LATEST Jan 6, 2020
liveness.py Happy new year! Jan 3, 2020
mypy.ini Split adviser implementation for extensibility and testing Nov 12, 2019
requirements.txt Provide an option to configure pipeline via file/dict Dec 3, 2019
setup.py Random walk and initial configuration change Jan 6, 2020
thoth-adviser Initial template for implementation Mar 2, 2018

README.rst

Thoth Adviser

A recommendation engine and software stack generation for project Thoth.

There are the following main goals of thoth-adviser (as of now):

  1. Provide a tool that can compute recommendations in project Thoth.
  2. Check provenance of installed packages (which package source indexes are used - this is not guaranteed by pip nor Pipenv).
  3. A tool called "Dependency Monkey" that generates all the possible software stacks for a project respecting dependency resolution.

To interact with a deployed Thoth, you can use the Thamos CLI.

Dependency Monkey

Dependency Monkey is a functionality that allows you to generate all the possible software stacks for your project. Given the input (direct dependencies of your project), Dependency Monkey creates N-ary dependency graph (as described above) stating all the dependencies of your direct dependencies/libraries as well as dependencies of all the transitive dependencies in your application stack.

The primary use-case for Dependnecy Monkey is to generate software stacks that are subsequently validated and scored in the Amun service. Simply when generating all the possible software stacks, we can find the best software stack for an application by validating it in a CI (or Amun in case of Thoth), running the application in the specific runtime environment (e.g. Fedora 28 with installed native packages - RPMs) on some specific hardware configuration. Generating and scoring all the possible software stacks is, however, most often not doable in a reasonable time. For this purpose, Dependency Monkey can create a sample of software stacks (see the distribution and seed parameters) that can be taken as representatives. These representatives are scored and aggregated data are used for predicting the best application stack (again, generated and run through CI/Amun to make predictions more accurate by learning over time).

See :ref:`dependency_monkey` for more info.

Advises and Recommendations

In Thoth's terminology, advises and recommendations are the same. Based on aggregated knowledge stored in the graph database, provide the best application stack with reasoning on why the given software stack is used. There is reused the N-ary dependency graph implementation stated above to compute possible candidates of software stacks and based on data aggregated, there is performed scoring of software stacks based on solely package-level data (e.g. the given package cannot be installed into the given runtime environment) or software stack information - the combination of packages cannot be assembled together or there were spotted issues when the same packages were used together in some specific versions.

Provenance Checks

As Thoth aggregates information about packages available, it can verify a user's stack against its knowledge base. See :ref:`provenance_checks` for more info.

Package source configuration

When Thoth is deployed in your infrastracture that restricts packages installed to only trusted package source indexes, you can disable untrusted package source indexes by setting THOTH_WHITELISTED_SOURCES environment variable. This variable holds a comma separated list of URLs pointing to whitelisted package source indexes respecting PEP-0503 standard (the URL is with the /simple suffix).

This environment variable is automatically fed from Thoth's graph database in a deployment. This way Thoth's operator has full control on what package source indexes which are used by users of Thoth.

Installation and deployment

Adviser is built using OpenShift Source-to-Image and deployed automatically with Thoth's deployment playbooks available in the core repository.

In a Thoth deployment, adviser is run based on requests comming to the user API - each deployed adviser is run per a user request. You can run adviser locally as well by installing it and using its command line interface:

pip3 install thoth-adviser
thoth-adviser --help
# Or use git repo directly for the latest code:
# pip3 install git+https://github.com/thoth-station/adviser

When thoth-adviser is scheduled in a deployment, it is actually executed as a CLI with arguments passed via environment variables.

See thoth-storages repository repository on how to run Thoth's knowledge graph locally and example notebooks for experiments.

Adviser also considers environment variable THOTH_ADVISER_BLOCKED_UNITS that states a comma separated list of pipeline units that should not be added to the pipeline. This can be handy if an issue with a unit arises in a deployment - Thoth operator can remove pipeline unit by adjusting adviser template and provide this configuration without a need to deploy a new version of adviser.

Running adviser locally

Often it is useful to run adviser locally to experiment or verify your changes in implementation. You can do so easily by running:

pipenv install
PYTHONPATH=. pipenv run ./thoth-adviser --help

This command will run adviser locally - adviser will try to connect to a local PostgreSQL instance and compute recommendations. Browse docs here to see how to setup a local PostgreSQL instance. Also, follow the developer's guide to get more information about developer's setup.

You can’t perform that action at this time.