Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bigquery template can't create dataset, table, and view at the same time #98

Closed
marko7460 opened this issue Jun 4, 2019 · 0 comments
Closed
Assignees
Labels
bug Something isn't working DM Deployment Manager CFT

Comments

@marko7460
Copy link
Contributor

The following template is trying to create dataset, table, and view at the same time.

imports:
  - path: templates/bigquery/bigquery_dataset.py
    name: bigquery_dataset.py
  - path: templates/bigquery/bigquery_table.py
    name: bigquery_table.py

resources:
  - name: pun-bq-dataset
    type: bigquery_dataset.py
    properties:
      name: pun_bq_dataset
      location: US
      access:
        - role: OWNER
          userByEmail: <YOUR EMAIL>
  - name: pun-bq-table
    type: bigquery_table.py
    properties:
      name: pun_bq_table
      datasetId: $(ref.pun-bq-dataset.datasetId)
      schema:
        - name: firstname
          type: STRING
        - name: lastname
          type: STRING
        - name: age
          type: INTEGER
  - name: pun-bq-view
    type: bigquery_table.py
    metadata:
      dependsOn:
      - pun-bq-table
    properties:
      name: pun_bq_view
      datasetId: $(ref.pun-bq-dataset.datasetId)
      view:
        description: pun_bq_view
        useLegacySql: false
        query: "SELECT firstname, age FROM `<PROJECT_ID>.pun_bq_dataset.pun_bq_table`"

I'm not sure if this is the template issue or API issue but 75% of the time the deployment fails with error:

ERROR: (gcloud.deployment-manager.deployments.create) Error in Operation [operation-1559665247835-58a81dd10d54c-8bfdbf01-df6ae411]: errors:
- code: RESOURCE_ERROR
  location: /deployments/pun-bq-create1/resources/pun_bq_view
  message: '{"ResourceType":"bigquery.v2.table","ResourceErrorCode":"404","ResourceErrorMessage":{"code":404,"errors":[{"domain":"global","message":"Not
    found: Table <PROJECT_ID>:pun_bq_dataset.pun_bq_table","reason":"notFound"}],"message":"Not
    found: Table <PROJECT_ID>:pun_bq_dataset.pun_bq_table","statusMessage":"Not
    Found","requestPath":"https://www.googleapis.com/bigquery/v2/projects/<PROJECT_ID>/datasets/pun_bq_dataset/tables","httpMethod":"POST"}}'

When i run update after the failure all resources get created (missing view gets created on retry).

I also tried changing

    metadata:
      dependsOn:
      - pun-bq-table

to

    metadata:
      dependsOn:
      - <PROJECT_ID>:pun_bq_dataset.pun_bq_table

but that didn't help either. It seems that dependOn doesn't always behave as expected.

@aaron-lane aaron-lane added bug Something isn't working DM Deployment Manager CFT labels Jun 4, 2019
@ocsig ocsig closed this as completed Apr 17, 2023
Cloud Foundation Toolkit automation moved this from To do to Done Apr 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working DM Deployment Manager CFT
Projects
Development

No branches or pull requests

3 participants