Uncrecognized flag "input_pattern" error in 0.7.0 release.

[tneymanov]

0.7.0 seems to have introduced a bug for vcf_to_bq and vcf_to_bq_preprocessor pipelines with regards to input_pattern flag usage.

Operation Info:

• description: |-
  Stopped running "/opt/gcp_variant_transforms/bin/vcf_to_bq_preprocess --project my_project --input_pattern gs://my_bucket/
  test.g.vcf --report_path gs://my_bucket/test_preprocessor_report.tsv --resolved_headers_path gs://my_bucket/test_resolved_headers.vcf --report_all_conflicts true --temp_location gs://my_bucket/temp --job_name vcf-to
  -bigquery-preprocess-2019-04-15-13-39-38-850 --runner DataflowRunner --region us-central1": exit status 1: getopt: unrecognized option '--input_pattern'
  getopt: unrecognized option '--report_path'
  getopt: unrecognized option '--resolved_headers_path'
  getopt: unrecognized option '--report_all_conflicts'
  getopt: unrecognized option '--job_name'
  getopt: unrecognized option '--runner'
  getopt: unrecognized option '--region'
  details:
  '@type': type.googleapis.com/google.genomics.v2alpha1.ContainerStoppedEvent
  actionId: 4
  exitStatus: 1
  stderr: |
  getopt: unrecognized option '--input_pattern'
  getopt: unrecognized option '--report_path'
  getopt: unrecognized option '--resolved_headers_path'
  getopt: unrecognized option '--report_all_conflicts'
  getopt: unrecognized option '--job_name'
  getopt: unrecognized option '--runner'
  getopt: unrecognized option '--region'
  timestamp: '2019-04-15T18:41:54.155391Z'

• commands:
  - /opt/gcp_variant_transforms/bin/vcf_to_bq_preprocess
  - ...
  entrypoint: ''
  environment: {}
  flags: []
  imageUri: gcr.io/gcp-variant-transforms/gcp-variant-transforms
  labels: {}
  mounts: []
  name: ''
  pidNamespace: ''
  portMappings: {}

[tneymanov]

Ran the most primitive runs for both vcf_to_bq and vcf_to_bq_preprocess:

python2 -m gcp_variant_transforms.vcf_to_bq
--input_pattern gs://gcp-variant-transforms-testfiles/small_tests/valid-4.0.vcf
--output_table tural-test-runner:turalds.tempo001

python2 -m gcp_variant_transforms.vcf_to_bq_preprocess
--input_pattern gs://tural-test-runner/vcf_files/small/*.vcf
--report_path ../reportNEW
--report_all_conflicts True

Both finished without an issue.

Running integration tests for preprocessor rn.

[tneymanov]

Also the above were DirectRunner runs, going to try Docker run on top.

[kemp-google]

The issue is that the pipelines_runner.sh tool isn't looking for the old scripts as the first argument and invoking them. I missed this during my review of the script.

The script needs to be modified so that if one of the tools is the first argument (eg /opt/gcp_variant_transforms/bin/vcf_to_bq_preprocess), instead of proceeding it immediately invokes that tool with the remaining arguments.

Given that this will be a breaking change for all users, I have moved the latest tag back to 0.6.1 and we will do a patch release as soon as possible.

[tneymanov]

Fixed in Release v0.7.1.