[Bug]: Beam Yaml Jinja option may conflict with well known Pipeline option

[Abacn]

What happened?

Saying I have a Yaml pipeline has a Jinja vairable 'projectId', it may conflicts with the well known pipeline option project due to here

beam/sdks/python/apache_beam/yaml/main.py

Line 63 in 112685d

jinja_variable_parser = argparse.ArgumentParser()

jinja_variable_parser may "unambiguously" parsed "--project" as "--projectId", causing --project pipeline option treated as a jinja variable and removed from pipeline options passing to pipeline construction

This can cause weird error ValueError: Pipeline has validations errors: Missing required option: project. when submitting pipeline to Dataflow

We should set allow_abbrev to False. But this still doesn't fully avoid confliction when a jinja variable also named project

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[Abacn]

Fixed the case of a_jinja_variablel.startswith(a_pipeline_option). Re-open to track the scenario that jinja_variable sharing the same name with pipeline option

[krishiekrashie]

Hi, can I work on this issue? Please let me contribute

[Abacn]

Hi @krishiekrashie, thanks for your interests. The remaining part is a subtle issue and this bug is left for tracking purpose, to warn that users should not create jinja variable that has same name with well known pipeline options. Let me adjust the priority.