Policy tags removed by BigQueryInsertJobOperator in data-platform-foundations airflow DAG

[danieldeleo]

Both of the following BQ insert jobs in datapipeline_dc_tags.py strip the "customer_purchase" destination tables of their policy tags:

• cloud-foundation-fabric/blueprints/data-solutions/data-platform-foundations/demo/datapipeline_dc_tags.py

  Line 262 in c918cfc

  join_customer_purchase = BigQueryInsertJobOperator(

• cloud-foundation-fabric/blueprints/data-solutions/data-platform-foundations/demo/datapipeline_dc_tags.py

  Line 293 in c918cfc

  confidential_customer_purchase = BigQueryInsertJobOperator(

This occurs because the writeDisposition is set to "WRITE_TRUNCATE" which as the docs state:

• WRITE_TRUNCATE: If the table already exists, BigQuery overwrites the data, removes the constraints, and uses the schema from the query result.

This should be fixed because removing policy tags exposes sensitive data and removes the column-level security that policy tags enable.

[danieldeleo]

datapipeline_dc_tags_flex.py would require the same fix

[juliocc]

@lcaggio @iht can you take a look?

[danieldeleo]

In my tests, changing writeDisposition to WRITE_APPEND resolved the issue. I can submit PR fix tomorrow

[wiktorn]

There is BigQuery feature request for that. I agree that changing writeDisposition is a way forward for now.