[Bug] BQ is not applying policies to the same object at the same time

[Lana-hl]

Is this a new bug in dbt-bigquery?

• I believe this is a new bug in dbt-bigquery
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

BigQuery does not allow to apply policies to the same object at the same time, even if those policies have the same effect.
Error message: 500 IAM setPolicy failed for Dataset <>: There were concurrent policy changes.

Perhaps we would need to update how we apply and persist grants here?

Expected Behavior

Policies could be applied concurrently to the same object at the same time.

Steps To Reproduce

Only affects when BQ is running concurrent changes, and when running the grants separately and sequentially in the console returns a successful execution, which indicates that parallel invocations of these queries might be causing the issue.

Relevant log output

No response

Environment

- OS:
- Python:
- dbt-core:
- dbt-bigquery:

Additional Context

No response

[Fleid]

Hey there!
Could you help us reproduce the case? I'm wondering how you get concurrent changes here? Are you running multiple invocations of dbt in parallel?

[saahil-mehta]

Hi! So in our CI workflow, GitHub Actions, the dbt config has 32 threads. The error shows up when this is executed, however when threads are switched to 2 the error goes away. The multiple invocations are through GitHub Actions workflow when deploying to production environment. Best, Saahil.

…

On Thu, 11 Apr 2024 at 8:56 PM, Florian Eiden ***@***.***> wrote: Hey there! Could you help us reproduce the case? I'm wondering how you get concurrent changes here? Are you running multiple invocations of dbt in parallel? — Reply to this email directly, view it on GitHub <#1167 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEXPU4S3YOVMOF2A47F54GDY43TH3AVCNFSM6AAAAABFVV73I2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJQGQYTSMBQGE> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[Fleid]

Hi there, could you please add a minimum reproducible example to the issue, so we can investigate?
You can build that out of the project generated via dbt init, or by tinkering with jaffle shop.
Thank you!

[saahil-mehta]

Hello, I find it to be a complex setup that may not be easily replicated in a smaller project or a Minimal Reproducible Example. Nonetheless, the issue is clear enough to address without simplification. Here are the specifics:

• Our dbt instance consists of over 900 models, all of which compile and build successfully when executed from the core or cloud.
• The challenge surfaces when we commit our changes to our repository.
• The GitHub Actions workflow initiates by running dependencies and building the project. During this phase, we encounter a recurring error (IAM setPolicy failed for Dataset :_TABLE**: There were concurrent policy changes. Please retry the whole read-modify-write with exponential backoff. The request's ETag '\007\006\026\276r+\210+' did not match the current policy's ETag '\007\006\026\276r0\037***'.**).
• This error is linked to the model post-hooks where we specify the IAM grants:

  erp:
    staging:
      materialized: ephemeral
      dataset: erp
    output:
      materialized: table
      dataset: erp  
    +grants:
      roles/bigquery.dataViewer: [
        'xx@xx.com', 
        'xx@xx.com'
      ]
    +post-hook:
      - |
        {% if target.project == 'xx' %}
          GRANT `projects/xx/roles/bigquery.selfServe` ON SCHEMA `xx`.mart_erp TO "xx@xx.com", "xx@xx.com", "xx@xx.com"
        {% endif %}

• Below is our profiles.yml where we specify the threads:

  default:
    target: prod
    outputs:
      prod:
        type: bigquery
        method: service-account
        keyfile: xx
        project: xx
        schema: xx
        threads: 32  # this is the parameter causing the issue, if we switch this to 2, the issue goes away but the project takes 1 hour to build, in the current state it takes 9-10 mins
        location: europe-west1
        timeout_seconds: 300 ```

[Fleid]

So is the issue that the post-hook grants are operating at the schema level, but are executed at the model level?
So the same table will receive multiple concurrent grants from different unrelated model runs wrapping up?

[saahil-mehta]

So, for example : For a "role.xx" the post-hooks are trying to assign principals (users) to that same role (could be for 'n' tables). This updation must happen at once and be written back for the policy to be consistent. However, since we are working with 32 threads, we are trying to modify the policy status concurrently and the IAM scope encounters mismatch in state as several GRANTS are being processed concurrently and their state is volatile.

One thing to note here, even if we are granting the "role.xx" access to a dataset 'y' and that change is yet to be written back, if we try to allow the same role, access (parellelly) to dataset 'z', it would fail even when the changes don't concern each other. In the same manner, allowing access to the same dataset, concurrently, would fail.