BigQuery template has dataset/table race condition

[bohdanyurov-gl]

Creating dataset & tables in a single run has a high chance for "dataset not found error" to occur.

dependsOn does not help

[duizendstra]

Based on the code and the test file I concluded that the dependsOn should be in the properties as:

    properties:
      name: test_bq_table
      dependsOn:
        - test-bq-dataset

The documentation mentions that dependsOn should be added as metadata.

Adding it to the properties as shown above does get rid of the race condition, however, it seems wrong to add it to the properties

[ocsig]

Short answer

The BigQuery Table template is accepting a dependsOn value as a property and passes it to the resource metadata.

Other option is to reference the dataset as it is implemented in the example:
datasetId: $(ref.test-bq-dataset.datasetId)

This adds a direct dependency between the table and the dataset.

Long explanation

Unfortunately DM does not inherit the metadata value of a template to all the resoureces are created within the template and unfortunately the metadata is not accessible withing the template either, so it cannot do the inheritaion itself.