Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

st2 Pack Dependencies Not Resolved via st2-pack-install tool #4867

Open
danielburrell opened this issue Feb 13, 2020 · 2 comments
Open

st2 Pack Dependencies Not Resolved via st2-pack-install tool #4867

danielburrell opened this issue Feb 13, 2020 · 2 comments
Labels

Comments

@danielburrell
Copy link

@danielburrell danielburrell commented Feb 13, 2020

SUMMARY

The dependency resolution system between stackstorm packs (not python dependencies) which was recently implemented in st2 v3.2 (#4769 ) does not function when trying to build via st2-pack-install (#4256) utility mentioned in https://docs.stackstorm.com/install/k8s_ha.html#custom-st2-packs.

STACKSTORM VERSION

3.2dev

Steps to reproduce the problem

/opt/stackstorm/st2/bin/st2-pack-install -vv  https://github.com/stackstorm/stackstorm-ms

Expected Results

Expect to see the dependencies of stackstorm-ms being installed (excel, and powerpoint for example).

Actual Results

Only the stackstorm-ms pack is installed.

2020-02-13 17:44:53,902 INFO [-] Installing pack "https://github.com/StackStorm/stackstorm-ms"
2020-02-13 17:44:53,903 DEBUG [-] Popen(['git', 'clone', '-v', 'https://github.com/StackStorm/stackstorm-ms', '/root/6622c8eb7209250c62980483f40577e2'], cwd=/opt/stackstorm, universal_newlines=True, shell=None)
2020-02-13 17:44:55,509 DEBUG [-] Cmd(['git', 'clone', '-v', 'https://github.com/StackStorm/stackstorm-ms', '/root/6622c8eb7209250c62980483f40577e2'])'s unused stdout: 
2020-02-13 17:44:55,518 DEBUG [-] Popen(['git', 'cat-file', '--batch-check'], cwd=/root/6622c8eb7209250c62980483f40577e2, universal_newlines=False, shell=None)
2020-02-13 17:44:56,256 DEBUG [-] Popen(['git', 'branch', '-a', '--contains', '25089933a452dedeb5aa145d6716c0187917b854'], cwd=/root/6622c8eb7209250c62980483f40577e2, universal_newlines=False, shell=None)
2020-02-13 17:44:56,906 DEBUG [-] Popen(['git', 'checkout', '25089933a452dedeb5aa145d6716c0187917b854'], cwd=/root/6622c8eb7209250c62980483f40577e2, universal_newlines=False, shell=None)
2020-02-13 17:44:57,545 DEBUG [-] Popen(['git', 'branch', '-f', 'master', '25089933a452dedeb5aa145d6716c0187917b854'], cwd=/root/6622c8eb7209250c62980483f40577e2, universal_newlines=False, shell=None)
2020-02-13 17:44:58,292 DEBUG [-] Popen(['git', 'checkout', 'master'], cwd=/root/6622c8eb7209250c62980483f40577e2, universal_newlines=False, shell=None)
2020-02-13 17:44:58,940 DEBUG [-] Moving pack from /root/6622c8eb7209250c62980483f40577e2 to /opt/stackstorm/packs.
2020-02-13 17:44:58,942 DEBUG [-] Changing owner group of "/opt/stackstorm/packs/microsoft_test" directory to st2packs
2020-02-13 17:44:58,942 DEBUG [-] Entering st2common.util.green.run_command.
2020-02-13 17:44:58,942 DEBUG [-] env argument not provided. using process env (os.environ).
2020-02-13 17:44:58,942 DEBUG [-] Creating subprocess.
2020-02-13 17:44:58,948 DEBUG [-] Spawning timeout handler thread.
2020-02-13 17:44:58,949 DEBUG [-] Attaching to process.
2020-02-13 17:44:58,949 DEBUG [-] Using delayed stdout and stderr read mode, calling process.communicate()
2020-02-13 17:44:58,949 DEBUG [-] Starting process wait inside timeout handler.
2020-02-13 17:44:58,963 DEBUG [-] No timeout.
2020-02-13 17:44:58,963 DEBUG [-] Returning.
2020-02-13 17:44:58,966 INFO [-] Successfully installed pack "microsoft_test"
2020-02-13 17:44:58,966 INFO [-] Setting up virtualenv for pack "microsoft_test"
2020-02-13 17:44:58,968 DEBUG [-] Setting up virtualenv for pack "microsoft_test" (/opt/stackstorm/packs/microsoft_test)
2020-02-13 17:44:58,968 INFO [-] Virtualenv path "/opt/stackstorm/virtualenvs/microsoft_test" doesn't exist
2020-02-13 17:44:58,969 DEBUG [-] Creating virtualenv for pack "microsoft_test" in "/opt/stackstorm/virtualenvs/microsoft_test"
2020-02-13 17:44:58,969 DEBUG [-] Creating virtualenv in "/opt/stackstorm/virtualenvs/microsoft_test" using Python binary "/opt/stackstorm/st2/bin/python"
2020-02-13 17:44:58,969 DEBUG [-] Running command "/opt/stackstorm/st2/bin/virtualenv -p /opt/stackstorm/st2/bin/python --always-copy --no-download /opt/stackstorm/virtualenvs/microsoft_test" to create virtualenv.
2020-02-13 17:45:01,563 DEBUG [-] Installing base requirements
2020-02-13 17:45:01,563 DEBUG [-] Installing requirement six>=1.9.0,<2.0 with command /opt/stackstorm/virtualenvs/microsoft_test/bin/pip install six>=1.9.0,<2.0.
2020-02-13 17:45:02,018 DEBUG [-] No pack specific requirements found
2020-02-13 17:45:02,018 DEBUG [-] Changing owner group of "/opt/stackstorm/virtualenvs/microsoft_test" directory to st2packs
2020-02-13 17:45:02,018 DEBUG [-] Entering st2common.util.green.run_command.
2020-02-13 17:45:02,019 DEBUG [-] env argument not provided. using process env (os.environ).
2020-02-13 17:45:02,019 DEBUG [-] Creating subprocess.
2020-02-13 17:45:02,022 DEBUG [-] Spawning timeout handler thread.
2020-02-13 17:45:02,022 DEBUG [-] Attaching to process.
2020-02-13 17:45:02,022 DEBUG [-] Using delayed stdout and stderr read mode, calling process.communicate()
2020-02-13 17:45:02,022 DEBUG [-] Starting process wait inside timeout handler.
2020-02-13 17:45:02,044 DEBUG [-] No timeout.
2020-02-13 17:45:02,044 DEBUG [-] Returning.
2020-02-13 17:45:02,044 DEBUG [-] Virtualenv for pack "microsoft_test" successfully created in "/opt/stackstorm/virtualenvs/microsoft_test"
2020-02-13 17:45:02,044 INFO [-] Successfully set up virtualenv for pack "microsoft_test"

Thanks!

@armab armab changed the title St2 Pack Dependencies Not Resolved in HA builds via st2-pack-installed st2 Pack Dependencies Not Resolved via st2-pack-install tool Feb 13, 2020
@armab armab added the HA label Feb 13, 2020
@armab

This comment has been minimized.

Copy link
Member

@armab armab commented Feb 13, 2020

Thanks for the bug report. I've adjusted issue description a bit.

@armab

This comment has been minimized.

Copy link
Member

@armab armab commented Feb 13, 2020

After looking a bit closer, it turns out it's not possible to support pack dependencies via st2-pack-install tool at the moment.

st2-pack-install helper utility was designed to deploy the st2 packs, their virtualenvs without fully running st2 system and backends like DB/MQ.

While original st2 pack install in new version relies on several Orquesta workflows under the hood to gather the list, detect conflicts, download pack dependencies, meaning it requires full st2 engine up & running.

To support pack dependencies via helper st2-pack-install utility #4769 original implementation of pack dependencies needs a rewrite: replace Orquesta workflow with the pure Python logic.

@armab armab added refactor and removed good first issue labels Feb 13, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.