Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hailctl dataproc start fails in hail 0.2.129 #14452

Closed
ehigham opened this issue Apr 9, 2024 · 2 comments · Fixed by #14453
Closed

hailctl dataproc start fails in hail 0.2.129 #14452

ehigham opened this issue Apr 9, 2024 · 2 comments · Fixed by #14453
Assignees
Labels

Comments

@ehigham
Copy link
Collaborator

ehigham commented Apr 9, 2024

What happened?

hail-0.2.129-py3-none-any.whl bundled a version of hailtop/hailctl/deploy.yaml that was intended for internal testing only. This file provides configuration variables for hailctl. The file in 0.2.129 pointed to cloud resources in gs://hail-30-day/ that cause commands like hailctl dataproc start to fail due to one of the following:

  • the user does not have access to gs://hail-30-day, or
  • the resources have been deleted according to the bucket's 30-day lifecycle policy.

Version

0.2.129

Relevant log output

No response

@ehigham ehigham added the needs-triage A brand new issue that needs triaging. label Apr 9, 2024
@ehigham ehigham self-assigned this Apr 9, 2024
@ehigham
Copy link
Collaborator Author

ehigham commented Apr 9, 2024

This will be fixed in the next release.
If you can't wait until then and you're comfortable patching this yourself, replace the contents of the affected file with the following:

dataproc:
  init_notebook.py: gs://hail-common/hailctl/dataproc/0.2.129/init_notebook.py
  vep-GRCh37.sh: gs://hail-common/hailctl/dataproc/0.2.129/vep-GRCh37.sh
  vep-GRCh38.sh: gs://hail-common/hailctl/dataproc/0.2.129/vep-GRCh38.sh
  wheel: gs://hail-common/hailctl/dataproc/0.2.129/hail-0.2.129-py3-none-any.whl
  pip_dependencies: aiodns==2.0.0|aiohttp==3.9.3|aiosignal==1.3.1|async-timeout==4.0.3|attrs==23.2.0|avro==1.11.3|azure-common==1.1.28|azure-core==1.30.1|azure-identity==1.15.0|azure-mgmt-core==1.4.0|azure-mgmt-storage==20.1.0|azure-storage-blob==12.19.0|bokeh==3.3.4|boto3==1.34.55|botocore==1.34.55|cachetools==5.3.3|certifi==2024.2.2|cffi==1.16.0|charset-normalizer==3.3.2|click==8.1.7|commonmark==0.9.1|contourpy==1.2.0|cryptography==42.0.5|decorator==4.4.2|deprecated==1.2.14|dill==0.3.8|frozenlist==1.4.1|google-auth==2.28.1|google-auth-oauthlib==0.8.0|humanize==1.1.0|idna==3.6|isodate==0.6.1|janus==1.0.0|jinja2==3.1.3|jmespath==1.0.1|jproperties==2.1.1|markupsafe==2.1.5|msal==1.27.0|msal-extensions==1.1.0|msrest==0.7.1|multidict==6.0.5|nest-asyncio==1.6.0|numpy==1.26.4|oauthlib==3.2.2|orjson==3.9.10|packaging==23.2|pandas==2.2.1|parsimonious==0.10.0|pillow==10.2.0|plotly==5.19.0|portalocker==2.8.2|py4j==0.10.9.5|pyasn1==0.5.1|pyasn1-modules==0.3.0|pycares==4.4.0|pycparser==2.21|pygments==2.17.2|pyjwt[crypto]==2.8.0|python-dateutil==2.9.0.post0|python-json-logger==2.0.7|pytz==2024.1|pyyaml==6.0.1|regex==2023.12.25|requests==2.31.0|requests-oauthlib==1.3.1|rich==12.6.0|rsa==4.9|s3transfer==0.10.0|scipy==1.11.4|six==1.16.0|sortedcontainers==2.4.0|tabulate==0.9.0|tenacity==8.2.3|tornado==6.4|typer==0.9.0|typing-extensions==4.10.0|tzdata==2024.1|urllib3==1.26.18|uvloop==0.19.0;sys_platform!="win32"|wrapt==1.16.0|xyzservices==2023.10.1|yarl==1.9.4|

@ehigham ehigham added bug and removed needs-triage A brand new issue that needs triaging. labels Apr 9, 2024
@ehigham
Copy link
Collaborator Author

ehigham commented Apr 9, 2024

Since a89d64a

ehigham added a commit to ehigham/hail that referenced this issue Apr 9, 2024
In a89d64a, I modified `build.yaml` to release the wheel we had already
built and tested. Unbeknownst to me was that we rebuild the wheel with a
different version of `hail/python/hailtop/hailctl/deploy.yaml` and
releasing the version used for testing borked `hailctl dataproc`
commands.

To fix this, we'll rebuild the wheel but use the `jar` we've already
built and tested. This is safe to do as far as I know because we don't
bundle any information into the jar that depends on the make flag
`DEPLOY_REMOTE`.

Fixes: hail-is#14452
hail-ci-robot pushed a commit that referenced this issue Apr 10, 2024
In a89d64a, I modified `build.yaml` to release the wheel we had already
built and tested. Unbeknownst to me was that we rebuild the wheel with a
different version of `hail/python/hailtop/hailctl/deploy.yaml` and
releasing the version used for testing borked `hailctl dataproc`
commands.

To fix this, we'll rebuild the wheel but use the `jar` we've already
built and tested. This is safe to do as far as I know because we don't
bundle any information into the jar that depends on the make flag
`DEPLOY_REMOTE`.

Fixes: #14452
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant