Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[hailctl] make the output of dry-run copy-pasteable #14127

Merged
merged 1 commit into from Jan 9, 2024
Merged

[hailctl] make the output of dry-run copy-pasteable #14127

merged 1 commit into from Jan 9, 2024

Conversation

danking
Copy link
Collaborator

@danking danking commented Jan 5, 2024 •
edited

Here's a diff of hailctl dataproc start foo --dry-run on main and on this branch. Notice that the properties and metadata arguments gain a leading and trailing single quote. This ensure that things like sys_platform!="win32" are properly transmitted.

In start.py we just use exec-style invocation, so there's no equivalent issue.

7c7
<     --properties=^|||^spark:spark.task.maxFailures=20|||spark:spark.driver.extraJavaOptions=-Xss4M|||spark:spark.executor.extraJavaOptions=-Xss4M|||spark:spark.speculation=true|||hdfs:dfs.replication=1|||dataproc:dataproc.logging.stackdriver.enable=false|||dataproc:dataproc.monitoring.stackdriver.enable=false|||spark:spark.driver.memory=36g|||yarn:yarn.nodemanager.resource.memory-mb=29184|||yarn:yarn.scheduler.maximum-allocation-mb=14592|||spark:spark.executor.cores=4|||spark:spark.executor.memory=5837m|||spark:spark.executor.memoryOverhead=8755m|||spark:spark.memory.storageFraction=0.2|||spark:spark.executorEnv.HAIL_WORKER_OFF_HEAP_MEMORY_PER_CORE_MB=3648 \
---
>     '--properties=^|||^spark:spark.task.maxFailures=20|||spark:spark.driver.extraJavaOptions=-Xss4M|||spark:spark.executor.extraJavaOptions=-Xss4M|||spark:spark.speculation=true|||hdfs:dfs.replication=1|||dataproc:dataproc.logging.stackdriver.enable=false|||dataproc:dataproc.monitoring.stackdriver.enable=false|||spark:spark.driver.memory=36g|||yarn:yarn.nodemanager.resource.memory-mb=29184|||yarn:yarn.scheduler.maximum-allocation-mb=14592|||spark:spark.executor.cores=4|||spark:spark.executor.memory=5837m|||spark:spark.executor.memoryOverhead=8755m|||spark:spark.memory.storageFraction=0.2|||spark:spark.executorEnv.HAIL_WORKER_OFF_HEAP_MEMORY_PER_CORE_MB=3648' \
9c9
<     --metadata=^|||^WHEEL=gs://hail-30-day/hailctl/dataproc/dking-dev/0.2.126-a51eabd65859/hail-0.2.126-py3-none-any.whl|||PKGS=aiodns==2.0.0|aiohttp==3.9.1|aiosignal==1.3.1|async-timeout==4.0.3|attrs==23.1.0|avro==1.11.3|azure-common==1.1.28|azure-core==1.29.5|azure-identity==1.15.0|azure-mgmt-core==1.4.0|azure-mgmt-storage==20.1.0|azure-storage-blob==12.19.0|bokeh==3.3.1|boto3==1.33.1|botocore==1.33.1|cachetools==5.3.2|certifi==2023.11.17|cffi==1.16.0|charset-normalizer==3.3.2|click==8.1.7|commonmark==0.9.1|contourpy==1.2.0|cryptography==41.0.7|decorator==4.4.2|deprecated==1.2.14|dill==0.3.7|frozenlist==1.4.0|google-auth==2.23.4|google-auth-oauthlib==0.8.0|humanize==1.1.0|idna==3.6|isodate==0.6.1|janus==1.0.0|jinja2==3.1.2|jmespath==1.0.1|jproperties==2.1.1|markupsafe==2.1.3|msal==1.25.0|msal-extensions==1.0.0|msrest==0.7.1|multidict==6.0.4|nest-asyncio==1.5.8|numpy==1.26.2|oauthlib==3.2.2|orjson==3.9.10|packaging==23.2|pandas==2.1.3|parsimonious==0.10.0|pillow==10.1.0|plotly==5.18.0|portalocker==2.8.2|protobuf==3.20.2|py4j==0.10.9.5|pyasn1==0.5.1|pyasn1-modules==0.3.0|pycares==4.4.0|pycparser==2.21|pygments==2.17.2|pyjwt[crypto]==2.8.0|python-dateutil==2.8.2|python-json-logger==2.0.7|pytz==2023.3.post1|pyyaml==6.0.1|regex==2023.10.3|requests==2.31.0|requests-oauthlib==1.3.1|rich==12.6.0|rsa==4.9|s3transfer==0.8.0|scipy==1.11.4|six==1.16.0|sortedcontainers==2.4.0|tabulate==0.9.0|tenacity==8.2.3|tornado==6.3.3|typer==0.9.0|typing-extensions==4.8.0|tzdata==2023.3|urllib3==1.26.18|uvloop==0.19.0;sys_platform!="win32"|wrapt==1.16.0|xyzservices==2023.10.1|yarl==1.9.3 \
---
>     '--metadata=^|||^WHEEL=gs://hail-30-day/hailctl/dataproc/dking-dev/0.2.126-a51eabd65859/hail-0.2.126-py3-none-any.whl|||PKGS=aiodns==2.0.0|aiohttp==3.9.1|aiosignal==1.3.1|async-timeout==4.0.3|attrs==23.1.0|avro==1.11.3|azure-common==1.1.28|azure-core==1.29.5|azure-identity==1.15.0|azure-mgmt-core==1.4.0|azure-mgmt-storage==20.1.0|azure-storage-blob==12.19.0|bokeh==3.3.1|boto3==1.33.1|botocore==1.33.1|cachetools==5.3.2|certifi==2023.11.17|cffi==1.16.0|charset-normalizer==3.3.2|click==8.1.7|commonmark==0.9.1|contourpy==1.2.0|cryptography==41.0.7|decorator==4.4.2|deprecated==1.2.14|dill==0.3.7|frozenlist==1.4.0|google-auth==2.23.4|google-auth-oauthlib==0.8.0|humanize==1.1.0|idna==3.6|isodate==0.6.1|janus==1.0.0|jinja2==3.1.2|jmespath==1.0.1|jproperties==2.1.1|markupsafe==2.1.3|msal==1.25.0|msal-extensions==1.0.0|msrest==0.7.1|multidict==6.0.4|nest-asyncio==1.5.8|numpy==1.26.2|oauthlib==3.2.2|orjson==3.9.10|packaging==23.2|pandas==2.1.3|parsimonious==0.10.0|pillow==10.1.0|plotly==5.18.0|portalocker==2.8.2|protobuf==3.20.2|py4j==0.10.9.5|pyasn1==0.5.1|pyasn1-modules==0.3.0|pycares==4.4.0|pycparser==2.21|pygments==2.17.2|pyjwt[crypto]==2.8.0|python-dateutil==2.8.2|python-json-logger==2.0.7|pytz==2023.3.post1|pyyaml==6.0.1|regex==2023.10.3|requests==2.31.0|requests-oauthlib==1.3.1|rich==12.6.0|rsa==4.9|s3transfer==0.8.0|scipy==1.11.4|six==1.16.0|sortedcontainers==2.4.0|tabulate==0.9.0|tenacity==8.2.3|tornado==6.3.3|typer==0.9.0|typing-extensions==4.8.0|tzdata==2023.3|urllib3==1.26.18|uvloop==0.19.0;sys_platform!="win32"|wrapt==1.16.0|xyzservices==2023.10.1|yarl==1.9.3' \

@danking
[hailctl] make the output of dry-run copy-pasteable
a53bf4a 
Here's a diff of `hailctl dataproc start foo --dry-run` on main and on this branch. Notice that the
properties and metadata arguments gain a leading and trailing single quote. This ensure that things
like `sys_platform!="win32"` are properly transmitted.

In `start.py` we just use exec-style invocation, so there's no equivalent issue.

```
7c7
<     --properties=^|||^spark:spark.task.maxFailures=20|||spark:spark.driver.extraJavaOptions=-Xss4M|||spark:spark.executor.extraJavaOptions=-Xss4M|||spark:spark.speculation=true|||hdfs:dfs.replication=1|||dataproc:dataproc.logging.stackdriver.enable=false|||dataproc:dataproc.monitoring.stackdriver.enable=false|||spark:spark.driver.memory=36g|||yarn:yarn.nodemanager.resource.memory-mb=29184|||yarn:yarn.scheduler.maximum-allocation-mb=14592|||spark:spark.executor.cores=4|||spark:spark.executor.memory=5837m|||spark:spark.executor.memoryOverhead=8755m|||spark:spark.memory.storageFraction=0.2|||spark:spark.executorEnv.HAIL_WORKER_OFF_HEAP_MEMORY_PER_CORE_MB=3648 \
---
>     '--properties=^|||^spark:spark.task.maxFailures=20|||spark:spark.driver.extraJavaOptions=-Xss4M|||spark:spark.executor.extraJavaOptions=-Xss4M|||spark:spark.speculation=true|||hdfs:dfs.replication=1|||dataproc:dataproc.logging.stackdriver.enable=false|||dataproc:dataproc.monitoring.stackdriver.enable=false|||spark:spark.driver.memory=36g|||yarn:yarn.nodemanager.resource.memory-mb=29184|||yarn:yarn.scheduler.maximum-allocation-mb=14592|||spark:spark.executor.cores=4|||spark:spark.executor.memory=5837m|||spark:spark.executor.memoryOverhead=8755m|||spark:spark.memory.storageFraction=0.2|||spark:spark.executorEnv.HAIL_WORKER_OFF_HEAP_MEMORY_PER_CORE_MB=3648' \
9c9
<     --metadata=^|||^WHEEL=gs://hail-30-day/hailctl/dataproc/dking-dev/0.2.126-a51eabd65859/hail-0.2.126-py3-none-any.whl|||PKGS=aiodns==2.0.0|aiohttp==3.9.1|aiosignal==1.3.1|async-timeout==4.0.3|attrs==23.1.0|avro==1.11.3|azure-common==1.1.28|azure-core==1.29.5|azure-identity==1.15.0|azure-mgmt-core==1.4.0|azure-mgmt-storage==20.1.0|azure-storage-blob==12.19.0|bokeh==3.3.1|boto3==1.33.1|botocore==1.33.1|cachetools==5.3.2|certifi==2023.11.17|cffi==1.16.0|charset-normalizer==3.3.2|click==8.1.7|commonmark==0.9.1|contourpy==1.2.0|cryptography==41.0.7|decorator==4.4.2|deprecated==1.2.14|dill==0.3.7|frozenlist==1.4.0|google-auth==2.23.4|google-auth-oauthlib==0.8.0|humanize==1.1.0|idna==3.6|isodate==0.6.1|janus==1.0.0|jinja2==3.1.2|jmespath==1.0.1|jproperties==2.1.1|markupsafe==2.1.3|msal==1.25.0|msal-extensions==1.0.0|msrest==0.7.1|multidict==6.0.4|nest-asyncio==1.5.8|numpy==1.26.2|oauthlib==3.2.2|orjson==3.9.10|packaging==23.2|pandas==2.1.3|parsimonious==0.10.0|pillow==10.1.0|plotly==5.18.0|portalocker==2.8.2|protobuf==3.20.2|py4j==0.10.9.5|pyasn1==0.5.1|pyasn1-modules==0.3.0|pycares==4.4.0|pycparser==2.21|pygments==2.17.2|pyjwt[crypto]==2.8.0|python-dateutil==2.8.2|python-json-logger==2.0.7|pytz==2023.3.post1|pyyaml==6.0.1|regex==2023.10.3|requests==2.31.0|requests-oauthlib==1.3.1|rich==12.6.0|rsa==4.9|s3transfer==0.8.0|scipy==1.11.4|six==1.16.0|sortedcontainers==2.4.0|tabulate==0.9.0|tenacity==8.2.3|tornado==6.3.3|typer==0.9.0|typing-extensions==4.8.0|tzdata==2023.3|urllib3==1.26.18|uvloop==0.19.0;sys_platform!="win32"|wrapt==1.16.0|xyzservices==2023.10.1|yarl==1.9.3 \
---
>     '--metadata=^|||^WHEEL=gs://hail-30-day/hailctl/dataproc/dking-dev/0.2.126-a51eabd65859/hail-0.2.126-py3-none-any.whl|||PKGS=aiodns==2.0.0|aiohttp==3.9.1|aiosignal==1.3.1|async-timeout==4.0.3|attrs==23.1.0|avro==1.11.3|azure-common==1.1.28|azure-core==1.29.5|azure-identity==1.15.0|azure-mgmt-core==1.4.0|azure-mgmt-storage==20.1.0|azure-storage-blob==12.19.0|bokeh==3.3.1|boto3==1.33.1|botocore==1.33.1|cachetools==5.3.2|certifi==2023.11.17|cffi==1.16.0|charset-normalizer==3.3.2|click==8.1.7|commonmark==0.9.1|contourpy==1.2.0|cryptography==41.0.7|decorator==4.4.2|deprecated==1.2.14|dill==0.3.7|frozenlist==1.4.0|google-auth==2.23.4|google-auth-oauthlib==0.8.0|humanize==1.1.0|idna==3.6|isodate==0.6.1|janus==1.0.0|jinja2==3.1.2|jmespath==1.0.1|jproperties==2.1.1|markupsafe==2.1.3|msal==1.25.0|msal-extensions==1.0.0|msrest==0.7.1|multidict==6.0.4|nest-asyncio==1.5.8|numpy==1.26.2|oauthlib==3.2.2|orjson==3.9.10|packaging==23.2|pandas==2.1.3|parsimonious==0.10.0|pillow==10.1.0|plotly==5.18.0|portalocker==2.8.2|protobuf==3.20.2|py4j==0.10.9.5|pyasn1==0.5.1|pyasn1-modules==0.3.0|pycares==4.4.0|pycparser==2.21|pygments==2.17.2|pyjwt[crypto]==2.8.0|python-dateutil==2.8.2|python-json-logger==2.0.7|pytz==2023.3.post1|pyyaml==6.0.1|regex==2023.10.3|requests==2.31.0|requests-oauthlib==1.3.1|rich==12.6.0|rsa==4.9|s3transfer==0.8.0|scipy==1.11.4|six==1.16.0|sortedcontainers==2.4.0|tabulate==0.9.0|tenacity==8.2.3|tornado==6.3.3|typer==0.9.0|typing-extensions==4.8.0|tzdata==2023.3|urllib3==1.26.18|uvloop==0.19.0;sys_platform!="win32"|wrapt==1.16.0|xyzservices==2023.10.1|yarl==1.9.3' \
diff --git a/tmp/foo b/tmp/bar
index 0fa5acb944..5146b07c8b 100644
n```
@danking danking merged commit b543093 into hail-is:main Jan 9, 2024
8 checks passed
iris-garden pushed a commit to iris-garden/hail that referenced this pull request Jan 9, 2024
@danking @iris-garden
[hailctl] make the output of dry-run copy-pasteable (hail-is#14127)
6670f70 
Here's a diff of `hailctl dataproc start foo --dry-run` on main and on
this branch. Notice that the properties and metadata arguments gain a
leading and trailing single quote. This ensure that things like
`sys_platform!="win32"` are properly transmitted.

In `start.py` we just use exec-style invocation, so there's no
equivalent issue.

```
7c7
<     --properties=^|||^spark:spark.task.maxFailures=20|||spark:spark.driver.extraJavaOptions=-Xss4M|||spark:spark.executor.extraJavaOptions=-Xss4M|||spark:spark.speculation=true|||hdfs:dfs.replication=1|||dataproc:dataproc.logging.stackdriver.enable=false|||dataproc:dataproc.monitoring.stackdriver.enable=false|||spark:spark.driver.memory=36g|||yarn:yarn.nodemanager.resource.memory-mb=29184|||yarn:yarn.scheduler.maximum-allocation-mb=14592|||spark:spark.executor.cores=4|||spark:spark.executor.memory=5837m|||spark:spark.executor.memoryOverhead=8755m|||spark:spark.memory.storageFraction=0.2|||spark:spark.executorEnv.HAIL_WORKER_OFF_HEAP_MEMORY_PER_CORE_MB=3648 \
---
>     '--properties=^|||^spark:spark.task.maxFailures=20|||spark:spark.driver.extraJavaOptions=-Xss4M|||spark:spark.executor.extraJavaOptions=-Xss4M|||spark:spark.speculation=true|||hdfs:dfs.replication=1|||dataproc:dataproc.logging.stackdriver.enable=false|||dataproc:dataproc.monitoring.stackdriver.enable=false|||spark:spark.driver.memory=36g|||yarn:yarn.nodemanager.resource.memory-mb=29184|||yarn:yarn.scheduler.maximum-allocation-mb=14592|||spark:spark.executor.cores=4|||spark:spark.executor.memory=5837m|||spark:spark.executor.memoryOverhead=8755m|||spark:spark.memory.storageFraction=0.2|||spark:spark.executorEnv.HAIL_WORKER_OFF_HEAP_MEMORY_PER_CORE_MB=3648' \
9c9
<     --metadata=^|||^WHEEL=gs://hail-30-day/hailctl/dataproc/dking-dev/0.2.126-a51eabd65859/hail-0.2.126-py3-none-any.whl|||PKGS=aiodns==2.0.0|aiohttp==3.9.1|aiosignal==1.3.1|async-timeout==4.0.3|attrs==23.1.0|avro==1.11.3|azure-common==1.1.28|azure-core==1.29.5|azure-identity==1.15.0|azure-mgmt-core==1.4.0|azure-mgmt-storage==20.1.0|azure-storage-blob==12.19.0|bokeh==3.3.1|boto3==1.33.1|botocore==1.33.1|cachetools==5.3.2|certifi==2023.11.17|cffi==1.16.0|charset-normalizer==3.3.2|click==8.1.7|commonmark==0.9.1|contourpy==1.2.0|cryptography==41.0.7|decorator==4.4.2|deprecated==1.2.14|dill==0.3.7|frozenlist==1.4.0|google-auth==2.23.4|google-auth-oauthlib==0.8.0|humanize==1.1.0|idna==3.6|isodate==0.6.1|janus==1.0.0|jinja2==3.1.2|jmespath==1.0.1|jproperties==2.1.1|markupsafe==2.1.3|msal==1.25.0|msal-extensions==1.0.0|msrest==0.7.1|multidict==6.0.4|nest-asyncio==1.5.8|numpy==1.26.2|oauthlib==3.2.2|orjson==3.9.10|packaging==23.2|pandas==2.1.3|parsimonious==0.10.0|pillow==10.1.0|plotly==5.18.0|portalocker==2.8.2|protobuf==3.20.2|py4j==0.10.9.5|pyasn1==0.5.1|pyasn1-modules==0.3.0|pycares==4.4.0|pycparser==2.21|pygments==2.17.2|pyjwt[crypto]==2.8.0|python-dateutil==2.8.2|python-json-logger==2.0.7|pytz==2023.3.post1|pyyaml==6.0.1|regex==2023.10.3|requests==2.31.0|requests-oauthlib==1.3.1|rich==12.6.0|rsa==4.9|s3transfer==0.8.0|scipy==1.11.4|six==1.16.0|sortedcontainers==2.4.0|tabulate==0.9.0|tenacity==8.2.3|tornado==6.3.3|typer==0.9.0|typing-extensions==4.8.0|tzdata==2023.3|urllib3==1.26.18|uvloop==0.19.0;sys_platform!="win32"|wrapt==1.16.0|xyzservices==2023.10.1|yarl==1.9.3 \
---
>     '--metadata=^|||^WHEEL=gs://hail-30-day/hailctl/dataproc/dking-dev/0.2.126-a51eabd65859/hail-0.2.126-py3-none-any.whl|||PKGS=aiodns==2.0.0|aiohttp==3.9.1|aiosignal==1.3.1|async-timeout==4.0.3|attrs==23.1.0|avro==1.11.3|azure-common==1.1.28|azure-core==1.29.5|azure-identity==1.15.0|azure-mgmt-core==1.4.0|azure-mgmt-storage==20.1.0|azure-storage-blob==12.19.0|bokeh==3.3.1|boto3==1.33.1|botocore==1.33.1|cachetools==5.3.2|certifi==2023.11.17|cffi==1.16.0|charset-normalizer==3.3.2|click==8.1.7|commonmark==0.9.1|contourpy==1.2.0|cryptography==41.0.7|decorator==4.4.2|deprecated==1.2.14|dill==0.3.7|frozenlist==1.4.0|google-auth==2.23.4|google-auth-oauthlib==0.8.0|humanize==1.1.0|idna==3.6|isodate==0.6.1|janus==1.0.0|jinja2==3.1.2|jmespath==1.0.1|jproperties==2.1.1|markupsafe==2.1.3|msal==1.25.0|msal-extensions==1.0.0|msrest==0.7.1|multidict==6.0.4|nest-asyncio==1.5.8|numpy==1.26.2|oauthlib==3.2.2|orjson==3.9.10|packaging==23.2|pandas==2.1.3|parsimonious==0.10.0|pillow==10.1.0|plotly==5.18.0|portalocker==2.8.2|protobuf==3.20.2|py4j==0.10.9.5|pyasn1==0.5.1|pyasn1-modules==0.3.0|pycares==4.4.0|pycparser==2.21|pygments==2.17.2|pyjwt[crypto]==2.8.0|python-dateutil==2.8.2|python-json-logger==2.0.7|pytz==2023.3.post1|pyyaml==6.0.1|regex==2023.10.3|requests==2.31.0|requests-oauthlib==1.3.1|rich==12.6.0|rsa==4.9|s3transfer==0.8.0|scipy==1.11.4|six==1.16.0|sortedcontainers==2.4.0|tabulate==0.9.0|tenacity==8.2.3|tornado==6.3.3|typer==0.9.0|typing-extensions==4.8.0|tzdata==2023.3|urllib3==1.26.18|uvloop==0.19.0;sys_platform!="win32"|wrapt==1.16.0|xyzservices==2023.10.1|yarl==1.9.3' \
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@daniel-goldstein daniel-goldstein daniel-goldstein approved these changes

Assignees

@daniel-goldstein daniel-goldstein

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@danking @daniel-goldstein