[Platform] Creating AWS provider fails on non-rep onprem (2.5.3.1 yugabundle) #7597
Comments
WesleyW
added a commit
that referenced
this issue
Mar 11, 2021
Summary: Updated the python3 requirements.txt and python2 requirements.txt so that no conflicting dependencies exist. Also cleaned up some python references so it's clear whether py2 or py3 is being used. Note - due to multiple python dependencies requiring the `cffi` package, the generated python tar.gz can only be used with the same python version that generated it. e.g. If it's packaged with py3.7, then only py3.7 can run it. Test Plan: Set up non-replicated install of platform on a new instance Create devops package locally with `devops/yb_release --force` and copy it over to new instance Run `devops/bin/install_python_requirements.sh --use_package` Try out create provider + universe flow Reviewers: sanketh, arnav, sb-yb, spotachev, daniel Reviewed By: daniel Subscribers: jenkins-bot, yugaware Differential Revision: https://phabricator.dev.yugabyte.com/D10881
WesleyW
added a commit
that referenced
this issue
Mar 11, 2021
Summary: D10881 did not make a final update to the frozen requirements, so this diff is adding that in. Test Plan: bin/freeze_python_requirements.sh Reviewers: sb-yb, spotachev, daniel Reviewed By: daniel Subscribers: yugaware Differential Revision: https://phabricator.dev.yugabyte.com/D10886
WesleyW
added a commit
that referenced
this issue
Mar 11, 2021
…plicated platform installs Summary: Updated the python3 requirements.txt and python2 requirements.txt so that no conflicting dependencies exist. Also cleaned up some python references so it's clear whether py2 or py3 is being used. Note - due to multiple python dependencies requiring the `cffi` package, the generated python tar.gz can only be used with the same python version that generated it. e.g. If it's packaged with py3.7, then only py3.7 can run it. Test Plan: Set up non-replicated install of platform on a new instance Create devops package locally with `devops/yb_release --force` and copy it over to new instance Run `devops/bin/install_python_requirements.sh --use_package` Try out create provider + universe flow Reviewers: sanketh, arnav, sb-yb, spotachev, daniel Reviewed By: daniel Subscribers: yugaware, jenkins-bot Differential Revision: https://phabricator.dev.yugabyte.com/D10887
WesleyW
added a commit
that referenced
this issue
Mar 11, 2021
Summary: D10887 did not make a final update to the frozen requirements, so this diff is adding that in. Test Plan: bin/freeze_python_requirements.sh Reviewers: sanketh, arnav, daniel Reviewed By: daniel Subscribers: yugaware Differential Revision: https://phabricator.dev.yugabyte.com/D10889
WesleyW
added a commit
that referenced
this issue
Mar 19, 2021
…installations Summary: For non-replicated installations, `bin/install_python_requirements.sh --create_package` will download all the python scripts that are added to PATH (i.e. ansible) first, and then download the libraries needed. To work around the use of namespace packages in the libaries (like for `google-api-core`), we add a custom startup script called `sitecustomize.py` for python (see https://docs.python.org/3/library/site.html) that adds the necessary packages to the path. Test Plan: Ran itest pointing to non-replicated setup. Reviewers: rtsisyk, daniel Reviewed By: daniel Subscribers: jenkins-bot, yugaware Differential Revision: https://phabricator.dev.yugabyte.com/D10985
WesleyW
added a commit
that referenced
this issue
Mar 20, 2021
…on-replicated installations Summary: For non-replicated installations, `bin/install_python_requirements.sh --create_package` will download all the python scripts that are added to PATH (i.e. ansible) first, and then download the libraries needed. To work around the use of namespace packages in the libaries (like for `google-api-core`), we add a custom startup script called `sitecustomize.py` for python (see https://docs.python.org/3/library/site.html) that adds the necessary packages to the path. Test Plan: Ran itest pointing to non-replicated setup. Reviewers: rtsisyk, daniel Reviewed By: daniel Subscribers: yugaware, jenkins-bot Differential Revision: https://phabricator.dev.yugabyte.com/D10999
WesleyW
added a commit
that referenced
this issue
Mar 26, 2021
…on-replicated installations Summary: For non-replicated installations, `bin/install_python_requirements.sh --create_package` will download all the python scripts that are added to PATH (i.e. ansible) first, and then download the libraries needed. To work around the use of namespace packages in the libaries (like for `google-api-core`), we add a custom startup script called `sitecustomize.py` for python (see https://docs.python.org/3/library/site.html) that adds the necessary packages to the path. Also squashed the following into this commit: D11010, D11025, D11034, D11050 Test Plan: Ran itest pointing to non-replicated setup. Reviewers: daniel, rtsisyk Reviewed By: rtsisyk Subscribers: jenkins-bot, yugaware Differential Revision: https://phabricator.dev.yugabyte.com/D11052
WesleyW
added a commit
that referenced
this issue
Apr 6, 2021
… non-replicated installations Summary: For non-replicated installations, `bin/install_python_requirements.sh --create_package` will download all the python scripts that are added to PATH (i.e. ansible) first, and then download the libraries needed. To work around the use of namespace packages in the libaries (like for `google-api-core`), we add a custom startup script called `sitecustomize.py` for python (see https://docs.python.org/3/library/site.html) that adds the necessary packages to the path. Also squashed the following into this commit: D11010, D11025, D11034, D11050, D11094 Test Plan: Ran itest pointing to non-replicated setup. Reviewers: rtsisyk, spotachev, steve.varnau, daniel Reviewed By: daniel Subscribers: yugaware, jenkins-bot Differential Revision: https://phabricator.dev.yugabyte.com/D11169
WesleyW
added a commit
that referenced
this issue
Apr 14, 2021
…in non-replicated installations Summary: For non-replicated installations, `bin/install_python_requirements.sh --create_package` will download all the python scripts that are added to PATH (i.e. ansible) first, and then download the libraries needed. To work around the use of namespace packages in the libaries (like for `google-api-core`), we add a custom startup script called `sitecustomize.py` for python (see https://docs.python.org/3/library/site.html) that adds the necessary packages to the path. This squashes D10985, D11010, D11025, D11034, D11050, D11093 Test Plan: Ran itest pointing to non-replicated setup. Reviewers: daniel, rtsisyk Reviewed By: rtsisyk Subscribers: yugaware, jenkins-bot Differential Revision: https://phabricator.dev.yugabyte.com/D11233
YintongMa
pushed a commit
to YintongMa/yugabyte-db
that referenced
this issue
May 26, 2021
…stalls Summary: Updated the python3 requirements.txt and python2 requirements.txt so that no conflicting dependencies exist. Also cleaned up some python references so it's clear whether py2 or py3 is being used. Note - due to multiple python dependencies requiring the `cffi` package, the generated python tar.gz can only be used with the same python version that generated it. e.g. If it's packaged with py3.7, then only py3.7 can run it. Test Plan: Set up non-replicated install of platform on a new instance Create devops package locally with `devops/yb_release --force` and copy it over to new instance Run `devops/bin/install_python_requirements.sh --use_package` Try out create provider + universe flow Reviewers: sanketh, arnav, sb-yb, spotachev, daniel Reviewed By: daniel Subscribers: jenkins-bot, yugaware Differential Revision: https://phabricator.dev.yugabyte.com/D10881
YintongMa
pushed a commit
to YintongMa/yugabyte-db
that referenced
this issue
May 26, 2021
Summary: D10881 did not make a final update to the frozen requirements, so this diff is adding that in. Test Plan: bin/freeze_python_requirements.sh Reviewers: sb-yb, spotachev, daniel Reviewed By: daniel Subscribers: yugaware Differential Revision: https://phabricator.dev.yugabyte.com/D10886
YintongMa
pushed a commit
to YintongMa/yugabyte-db
that referenced
this issue
May 26, 2021
…licated installations Summary: For non-replicated installations, `bin/install_python_requirements.sh --create_package` will download all the python scripts that are added to PATH (i.e. ansible) first, and then download the libraries needed. To work around the use of namespace packages in the libaries (like for `google-api-core`), we add a custom startup script called `sitecustomize.py` for python (see https://docs.python.org/3/library/site.html) that adds the necessary packages to the path. Test Plan: Ran itest pointing to non-replicated setup. Reviewers: rtsisyk, daniel Reviewed By: daniel Subscribers: jenkins-bot, yugaware Differential Revision: https://phabricator.dev.yugabyte.com/D10985
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Creating a AWS provider fails on non-replicated VM with the following error
Failed to execute task {"errorString":null,"providerUUID":"c2c93527-658f-4220-af43-fc5a3730ad3e","customPayload":"{"errorString":null,"providerUUID":"c2c93527-658f-4220-af43-fc5a3730ad3e","perRegionMetadata":{"us-west-2":{"vpcId":null,"vpcCidr":null,"azToSubnetIds":null,"subnetId":null,"customImageId":null,"customSecurityGroupId":null}},"keyPairName":null,"sshPrivateKeyContent":null,"sshUser":null,"airGapInstall":false,"sshPort":54422,"hostVpcId":null,"hostVpcRegion":null,"customHostCidrs":[],"destVpcId":null}"}, hit error java.lang.RuntimeException: YBCloud command network (bootstrap) failed to execute..
Stack trace:
opt/yugabyte/packages/yugabyte-2.5.3.1-b3/devops/venv/bin/python: can't find 'main' module in ''
2021-03-10 22:48:37,474 [INFO] from ShellProcessHandler in application-akka.actor.default-dispatcher-73 - Starting proc (abbrev cmd) - bin/ybcloud.sh gcp query current-host
2021-03-10 22:48:37,474 [DEBUG] from ShellProcessHandler in application-akka.actor.default-dispatcher-73 - Starting proc (full cmd) - 'bin/ybcloud.sh' 'gcp' 'query' 'current-host' - logging stdout=/tmp/shell_process_out4927130075606551311tmp, stderr=/tmp/shell_process_err6913270735519020552tmp
2021-03-10 22:48:37,520 [DEBUG] from ShellProcessHandler in application-akka.actor.default-dispatcher-73 - Proc stdout for 'bin/ybcloud.sh gcp query current-host' |
2021-03-10 22:48:37,520 [DEBUG] from ShellProcessHandler in application-akka.actor.default-dispatcher-73 - Proc stderr for 'bin/ybcloud.sh gcp query current-host' | which: no ybcloud.py in (/opt/yugabyte/packages/yugabyte-2.5.3.1-b3/devops/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin)
/opt/yugabyte/packages/yugabyte-2.5.3.1-b3/devops/venv/bin/python: can't find 'main' module in ''
2021-03-10 22:48:37,520 [INFO] from ShellProcessHandler in application-akka.actor.default-dispatcher-73 - Completed proc 'bin/ybcloud.sh gcp query current-host' status=failure code=1 [ 46 ms ]
2021-03-10 22:48:37,520 [ERROR] from DevopsBase in application-akka.actor.default-dispatcher-73 - which: no ybcloud.py in (/opt/yugabyte/packages/yugabyte-2.5.3.1-b3/devops/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin)
/opt/yugabyte/packages/yugabyte-2.5.3.1-b3/devops/venv/bin/python: can't find 'main' module in ''
Workaround as suggested by Wesley:
rm -rf /opt/yugabyte/packages/yugabyte-2.5.3.1-b3/devops/venv /opt/yugabyte/packages/yugabyte-2.5.3.1-b3/devops/python_virtual_env and run `/opt/yugabyte/packages/yugabyte-2.5.3.1-b3/devops/bin/install_python_requirements.sh
but still fails
/opt/yugabyte/packages/yugabyte-2.5.3.1-b3/devops/venv/bin/python: can't find 'main' module in ''
2021-03-11 00:14:43,278 [INFO] from ShellProcessHandler in TaskPool-CloudBootstrap(7ebbbc91-d744-43e3-a43d-87e0120014ad)-0 - Completed proc 'bin/ybcloud.sh aws network bootstrap {"errorString":null,"providerUUID":"7ebbbc91-d744-43e3-a43d-87e0120014ad","perRegionMetadata":{"us-west-2":{"vpcId":null,"vpcCidr":null,"azToSubnetIds":null,"subnetId":null,"customImageId":null,"customSecurityGroupId":null}},"keyPairName":null,"sshPrivateKeyContent":null,"sshUser":null,"airGapInstall":false,"sshPort":54422,"hostVpcId":null,"hostVpcRegion":null,"customHostCidrs":[],"destVpcId":null}' status=failure code=1 [ 51 ms ]
2021-03-11 00:14:43,278 [ERROR] from DevopsBase in TaskPool-CloudBootstrap(7ebbbc91-d744-43e3-a43d-87e0120014ad)-0 - which: no ybcloud.py in (/opt/yugabyte/packages/yugabyte-2.5.3.1-b3/devops/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin)
/opt/yugabyte/packages/yugabyte-2.5.3.1-b3/devops/venv/bin/python: can't find 'main' module in ''
2021-03-11 00:14:43,279 [ERROR] from SubTaskGroup in TaskPool-1 - Failed to execute task {"errorString":null,"providerUUID":"7ebbbc91-d744-43e3-a43d-87e0120014ad","customPayload":"{"errorString":null,"providerUUID":"7ebbbc91-d744-43e3-a43d-87e0120014ad","perRegionMetadata":{"us-west-2":{"vpcId":null,"vpcCidr":null,"azToSubnetIds":null,"subnetId":null,"customImageId":null,"customSecurityGroupId":null}},"keyPairName":null,"sshPrivateKeyContent":null,"sshUser":null,"airGapInstall":false,"sshPort":54422,"hostVpcId":null,"hostVpcRegion":null,"customHostCidrs":[],"destVpcId":null}"}, hit error java.lang.RuntimeException: YBCloud command network (bootstrap) failed to execute..
java.util.concurrent.ExecutionException: java.lang.RuntimeException: YBCloud command network (bootstrap) failed to execute.
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at com.yugabyte.yw.commissioner.SubTaskGroup.waitFor(SubTaskGroup.java:181)
at com.yugabyte.yw.commissioner.SubTaskGroupQueue.run(SubTaskGroupQueue.java:43)
at com.yugabyte.yw.commissioner.tasks.CloudBootstrap.run(CloudBootstrap.java:124)
at com.yugabyte.yw.commissioner.TaskRunner.run(TaskRunner.java:150)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: YBCloud command network (bootstrap) failed to execute.
at com.yugabyte.yw.commissioner.tasks.subtasks.cloud.CloudSetup.run(CloudSetup.java:45)
... 5 common frames omitted
2021-03-11 00:14:43,280 [ERROR] from SubTaskGroupQueue in TaskPool-1 - SubTaskGroup 'Create Cloud setup task : completed 0 out of 1 tasks.' waitFor() returned failed status.
2021-03-11 00:14:43,282 [ERROR] from TaskRunner in TaskPool-1 - Error running task
java.lang.RuntimeException: Create Cloud setup task : completed 0 out of 1 tasks. failed.
at com.yugabyte.yw.commissioner.SubTaskGroupQueue.run(SubTaskGroupQueue.java:56)
at com.yugabyte.yw.commissioner.tasks.CloudBootstrap.run(CloudBootstrap.java:124)
at com.yugabyte.yw.commissioner.TaskRunner.run(TaskRunner.java:150)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2021-03-11 00:14:43,283 [INFO] from TaskRunner in TaskPool-1 - Updating task [taskType : CloudBootstrap, taskState: Running] to new state Failure
The text was updated successfully, but these errors were encountered: