Dependent pack installation with "st2 pack install <pack>"

[sidkrishna]

Enhancement Request:

It would be good to if st2 pack install <pack1> could check for other pack dependencies for pack1 - say pack2, pack3 and implicitly also install them.

Example: If I create a custom pack eg. vslx and then pack uses actions from ansible pack (available in stackstorm exchange), then as part of the pack installation process for vslx, ansible pack can also be installed. The pack requirements can possibly be provided in the a file pack_requirements or any other better way of identification.

[arm4b]

+:100:

[nmaludy]

+1

[mickmcgrath13]

What about something like this: https://github.com/mickmcgrath13/bitovi_packs ?

[mickmcgrath13]

More specifically: https://github.com/bitovi-stackstorm-exchange/bitovi_packs/blob/master/actions/workflows/install.yaml
...?

[nmaludy]

@mickmcgrath13 that looks cool!!! mind creating a PR to implement that in the packs pack?

[jeking3]

I tried making my own pack that leverages the vSphere pack; when I try to run an action or workflow in the vSphere pack from my own pack, it fails claiming pyvmomi does not exist. This is a bit strange; I would expect that if execution crosses pack lines, the executions inside the dependent pack use the dependent pack's virtual environment, not the calling pack's. This does not appear to be the case. So any effort to allow for packs depending on other packs should address this issue. A good test for this is to try to call something in the vSphere pack from another path that does not set a requirements.txt dependency on pyvmomi. Should this be reported separately?

[LindsayHill]

@jeking3 if you have a workflow that calls an action in another pack - e.g. with vsphere.create_vm or whatever - then yes, that action will be executed using the vsphere pack's virtualenv, and will have access to whatever pip libs were installed in that pack.

This is standard functionality, and something I've done plenty of times, and it has always worked for me. If it's not working for you, then something else is going on. Probably worth posting on the support forum

[jeking3]

You are correct - I mistook the inventory not found error message for a class not found message. Thanks.

[LindsayHill]

Ah cool. Was worried something else was going on.

[namachieli]

My Use case: I have a custom internal pack that acts as a collection of my workflows and rules. One of those rules depends on a trigger in the zabbix pack. My custom pack fails to register when this trigger is missing. Because of how I iterate over my list of packs to be installed, its essentially a race condition.

Having a pack dependency of zabbix in my custom pack would fix this gracefully for me.

[jinpingh]

Copy https://github.com/StackStorm/discussions/issues/354 discussion to here:

Requirements:

1. If pack version is not specified, install latest tag version if tag is created, otherwise install from master as today.
2. If a pack foo dependents on other packs, for instance pack1 and pack2, these two packs should be installed.

Configuration for pack dependencies:

Pack dependency should be configured in pack and it will be saved to mongo db pack_d_b.

pack.yaml
dependencies:
  - jira=https://github.com/StackStorm-Exchange/pack1.git
  - web=https://github.com/StackStorm-Exchange/pack2.git

Design for Pack dependency management

Add new actions to packs.install workflow:
Get list of dependency packs that need to be installed after pack is successfully installed.
Run packs.install with dependency packs.

install.yaml
---
  chain:
    -
      name: "download pack"
      ref: "packs.download"
      parameters:
        packs: "{{packs}}"
        force: "{{force}}"
        python3: "{{python3}}"
      on-success: "make a prerun"
    -
      name: "make a prerun"
      ref: "packs.virtualenv_prerun"
      parameters:
        packs_status: "{{ __results['download pack'].result }}"
      on-success: "install pack dependencies"
    -
      name: "install pack dependencies"
      ref: "packs.setup_virtualenv"
      parameters:
        packs: "{{ __results['make a prerun'].result }}"
        env: "{{env}}"
        python3: "{{python3}}"
      on-success: "register pack"
    -
      name: "register pack"
      ref: "packs.load"
      parameters:
        register: "{{register}}"
        packs: "{{ __results['make a prerun'].result }}"
      on-success: "get dependency packs"
    -
      name: "get dependency packs"
      ref: "packs.get_dependency_packs"
      parameters:
        packs: "{{ __results['make a prerun'].result }}"
      on-success: "install dependency packs"
    -
      name: "install dependency packs"
      ref: "packs.install"
      parameters:
        packs: "{{__results['get dependency packs'].result}}"
        force: "{{force}}"
        python3: "{{python3}}"

  default: "download pack"

---

@m4dcoder

m4dcoder commented 4 days ago •
How is conflict detected and managed? Say a pack_a depends on pack_x version 1.1 and there is already a pack_b installed that depends on pack_x version 1.2. Or you are trying to install pack_b and it depends on pack_c and pack_d that depends different version of pack_x?

---

@jinpingh
Author
jinpingh commented 4 days ago
@m4dcoder During get dependency packs, if there are the same dependency packs with different version, the newer version should be installed and Warning message should be given. If newer version was installed already, only Warning message should be given. If we have to make discussion to pick up one version, hopefully newer version is better one.
Thanks for review it!

---

@armab
armab commented 2 hours ago •
Original feature request in st2 repo: #3336

How do we plan to install nested dependencies?
Say we install pack_a which depends on pack_b, while pack_b has pack_c in the dependencies list. Do we make a tree of dependencies/recursion or whatever?

Get list of dependency packs that need to be installed after pack is successfully installed.

I believe more correct implementation would be to get the pack dependencies installed first, because during the pack registration if pack_a tries to register content (say rules) that relies on ref.pack_b it won't pass validation before pack_b content is available.

---

@armab

armab commented 2 hours ago •
if there are the same dependency packs with different version, the newer version should be installed and Warning message should be given.

Installing new version by default when specific pack pins older version, while another pack pins specific newer version could be disastrous for production systems. Remember the case with our st2cicd server when we pin every st2 pack version and updating aws pack version from very old to new automatically would break workflows.

Would be nice to take ideas from other package managers.
Fail on conflict, but allow -y/yes flag to process such dependencies and install the latest one for someone who feels lucky.

WDYT?

---

@jinpingh
Author
jinpingh commented 1 hour ago •
I believe more correct implementation would be to get the pack dependencies installed first, because during the pack registration if pack_a tries to register content (say rules) that relies on ref.pack_b it won't pass validation before pack_b content is available.

@armab For dependency, from my point of view, we have to set up certain dependency rules. For example, if pack A dependency as: A->B->C, pack C dependency as: C->D->A. How to deal with it special if the pack dependencies installed first?

---

@jinpingh
Author
jinpingh commented 1 hour ago
Fail on conflict, but allow -y/yes flag to process such dependencies and install the latest one for someone who feels lucky.

Good point!

---

@armab

armab commented 14 minutes ago •
@jinpingh That's a dependency hell/circular dependency issue which I believe at best would be identified and errored.

See https://en.wikipedia.org/wiki/Dependency_hell which lists some of the recommendations and also highlights question that @m4dcoder raised.

From that link interesting point:

Solutions
Version numbering
A very common solution to this problem is to have a standardized numbering system, wherein software uses a specific number for each version (aka major version), and also a subnumber for each revision (aka minor version), e.g.: 10.1, or 5.7. The major version only changes when programs that used that version will no longer be compatible. The minor version might change with even a simple revision that does not prevent other software from working with it. In cases like this, software packages can then simply request a component that has a particular major version, and any minor version (greater than or equal to a particular minor version). As such, they will continue to work, and dependencies will be resolved successfully, even if the minor version changes. Semantic Versioning (aka "SemVer" [6]) is one example of an effort to generate a technical specification that employs specifically formatted numbers to create a software versioning scheme.

^^ That's a good idea of relying on versioning when doing any automated upgrades/dependency hell resolution and good stuff that will also encourage everyone to use correct semantic versioning, when "major version" should be always incremented when breaking changes were introduced. Right now contributors usually incorrectly bumping minor version X.Y.Z on breaking changes, instead of X.Y.Z major version.
We can rely on correct semantic version assumptions if doing any automated version upgrades and version resolutions.

Please copy entire discussion to open Github Issue #3336 to allow community give feedback as well.

[nmaludy]

Any thoughts on leveraging any existing package management systems to handle our dependencies?

I know this would be a big change, but could we potentially use something like pip and either PyPI or our own custom python package index to serve packages? This way we can leverage the "known" ways for handling dependencies within this ecosystem.

Same could be said for doing it with RPM/DEB. Bonus here is that we could have RPMs/DEBs depend on system level packages for things that require those pieces.

All of the points above are great points and my general question is: "Do we want to solve the problem ourselves, or piggy-back on an existing solution?"

For these scenarios i love a pros/cons list:

• Pros (using an existing package manager)
  • Able to utilize an existing ecosystem of convention and tooling
  • If we utilize pip then we can take advantage of StackStorm's already deep python heritage
  • if we utilize rpm/deb (yum/apt) we can take advantage of pre/post hooks and solve system dependencies (some system library/utility/etc)
• Cons (building our own)
  • Have to design and implement a system
  • Have to handle complex scenarios (a, b, c problem) as outlined in the discussion above
  • Using an existing system means we would need to have some sort of migration from our current pack layout into something new (would require new tooling, testing , etc)

[arm4b]

cc @mickmcgrath13 you might be interested as well ^^

[cognifloyd]

A lot of work is going on in the content-packaging realm with ansible/mazer as well.
I haven't looked at it close enough to evaluate feasibility, but we might be able to leverage parts of the ansible collections infrastructure for st2 packs.