Hitting ThrottlingException on GetWorkGroup with threads turned up

[dacreify]

Recently we started hitting ThrottlingException on GetWorkGroup calls at this spot in the code:

dbt-athena/dbt/adapters/athena/impl.py

Line 231 in a1b8db5

work_group = athena_client.get_work_group(WorkGroup=creds.work_group)

From CloudTrail it looks like dbt-athena winds up making a GetWorkGroup call for every model run. There's no documented quota for this call but obviously we're hitting one. Actual StartQueryExecution can go 20/second or burst to 80/second so GetWorkGroup definitely seems to be below that in any case.

Anyone else hit this? Can we cache the results of GetWorkGroup to avoid it?

[nicor88]

Thanks for your issue, caching could make sense. We can consider to put a caching wrapper on an inner function that contains

if creds.work_group:
            work_group = athena_client.get_work_group(WorkGroup=creds.work_group)
            output_location = (
                work_group.get("WorkGroup", {})
                .get("Configuration", {})
                .get("ResultConfiguration", {})
                .get("OutputLocation", None)
            )

            output_location_enforced = (
                work_group.get("WorkGroup", {}).get("Configuration", {}).get("EnforceWorkGroupConfiguration", False)
            )

            return output_location is not None and output_location_enforced
        else:
            return False

How many models do you have? I'm wondering which scale you have that lead you to hit this throttling.
We can even consider to add a retry wrapper with backoff on top of caching.

[nicor88]

@dacreify I forgot to come back to you, yes let's add a caching as you suggested. Do you want to take care of it or shall I? - if this is not enough, we will add a retry wrapper as well.

[nicor88]

@dacreify let's try this #596
if this caching is not enough, the only way that I can think of is to add a retry wrapper with backoff...
EDIT: the PR was merged, and it's in the main branch.
@dacreify and @juliansteger-sc could you try to install the adapter from a specific git commit and see if the issue is resolved?

[dacreify]

Thanks, @nicor88!

AWS acknowledged that this was a regression on our support request with them:

We saw an increase in throttling exceptions in the Athena service. Throttling exceptions occur when a user exceeds the limit for the number of Athena API calls allowed per second. Upon investigation, we discovered an error in how the API call rate was calculated. To address this, we have deployed a fix across all our regions on 03/12/2024. If you are still experiencing throttling exceptions or any related issues, we kindly ask you to reach out to us.

They told me what the rate limit on GetWorkGroup is but I think they're asking me not to disclose it. I will hazard to share that it is lower than the default for StartQueryExecution so this enhancement will still be a help for scaling.

[nicor88]

@dacreify Did you verify that the current caching implementation allow you to reduce the Throttling exceptions?

[dacreify]

Left results on the issue: #595 (comment)