how to pass the region argument to hailctl while running load_data_dataproc

[xshah]

I have gotten to the point where I am running this command

docker-compose exec pipeline-runner load_data_dataproc.sh $BUILD_VERSION $SAMPLE_TYPE $INDEX_NAME $GS_BUCKET $INPUT_FILE_PATH

and I am getting this error
'sudo docker compose exec pipeline-runner load_data_dataproc.sh 38 WGS techdev-j-jenn gs://seqr-v1/ master_data.vcf.gz

• BUILD_VERSION=38
• SAMPLE_TYPE=WGS
• INDEX_NAME=techdev-j-jenn
• GS_BUCKET=gs://seqr-v1/
• INPUT_FILE_PATH=master_data.vcf.gz
• case ${BUILD_VERSION} in
• FULL_BUILD_VERSION=GRCh38
• SOURCE_FILE=gs://seqr-v1/master_data.vcf.gz
• DEST_FILE=gs://seqr-v1/master_data.mt
• REFERENCE_DATA_BUCKET=gs://seqr-v1//reference_data/GRCh38
• cd /seqr-loading-pipelines/luigi_pipeline
• hailctl dataproc start --pkgs luigi,google-api-python-client --zone us-central1-b --vep GRCh38 --max-idle 30m --num-workers 2 --num-preemptible-workers 12 seqr-loading-cluster
  Traceback (most recent call last):
  File "/usr/local/bin/hailctl", line 8, in
  sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/hailtop/hailctl/main.py", line 107, in main
  cli.main(args)
  File "/usr/local/lib/python3.7/site-packages/hailtop/hailctl/dataproc/cli.py", line 124, in main
  jmp[args.module].main(args, pass_through_args))
  File "/usr/local/lib/python3.7/asyncio/base_events.py", line 573, in run_until_complete
  return future.result()
  File "/usr/local/lib/python3.7/site-packages/hailtop/hailctl/dataproc/start.py", line 290, in main
  raise RuntimeError("Could not determine dataproc region. Use --region argument to hailctl, or use gcloud config set dataproc/region <my-region> to set a default.")
  RuntimeError: Could not determine dataproc region. Use --region argument to hailctl, or use gcloud config set dataproc/region <my-region> to set a default.'

I have set the correct region using gcloud confit set command but still getting the same error

[hanars]

Just to clarify, did you run gcloud confit set or docker-compose exec pipeline-runner gcloud config set

[lauragails]

I tried to set the zone to us-east1-b any way I could find (including gcloud confit set, the latter wasn't working) and the zone flag keeps defaulting to "--zone=us-central1-b" in the load_data_dataproc.sh command.

Have you seen this before? Thank you!

[hanars]

Unfortunately the zone is hardcoded into that script. You can do the loading without using the load_data_dataproc.sh helper script by copying the commands the script is running, modifying them to change the zone as needed, and then running them using docker-compose exec. Alternatively, you can configure your GCP project to allow us-central1-b which might be easier

[lauragails]

Changing the zone, adding a region flag, and adding a few project specific flags resulted in a successfully created dataproc. I am still having issues with the next command that loads the data, but that is in another issue.

Thank you!