You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
I am unable to run queries using the astro.sql.transform decorator when they do not exist in the region where tmp_astro is created. In the below example error, I am running a query on a table in us-central1, when tmp_astro exists in europe-west2:
sqlalchemy.exc.DatabaseError: (google.cloud.bigquery.dbapi.exceptions.DatabaseError) 404 Not found: Dataset PROJECT_ID:tmp_astro was not found in location us-central1
Version
Astro: 1.3.0
OS: Debian
To Reproduce
Steps to reproduce the behavior:
Run the below DAG on using a TABLE_PATH in one region.
Then rerun it again using a different table path in another region.
"""
gcp_bq
DAG auto-generated by Astro Build.
"""
from airflow.decorators import dag
from astro import sql as aql
from astro.sql.table import Table
import pendulum
@aql.transform(conn_id="bq_fe_sa", task_id="cell_2")
def cell_2_func():
return """SELECT * FROM TABLE_PATH LIMIT 10;"""
@dag(
schedule_interval="0 0 * * *",
start_date=pendulum.from_format("2022-12-02", "YYYY-MM-DD").in_tz("America/Phoenix"),
)
def gcp_bq():
cell_2 = cell_2_func()
dag_obj = gcp_bq()
Expected behavior
I would expect the Astro SDK to make a tmp_astro dataset per region, and ensure that it is using/refencing the appropriate tmp schema based on the source table in the query.
Additional context
The BigQueryHook supports a location parameter, allowing you to specify the location/region for your query. I don't believe this can be set via a Connection parameter.
The text was updated successfully, but these errors were encountered:
# Description
## What is the current behavior?
Currently, we are assuming all the bigquery datasets to be in the
default region - `US`
If the user tries to access a preexisting table in a different region
say, `europe-west6` all the Temp tables will try to be created in `US`
the region, which will result in the error.
closes: #1369
## What is the new behavior?
1. The solution is to create `tmp_astro` region specific. But we cannot
do that since `<project_id>.<dataset_id>` is unique. So we are creating
a unique dataset id like - `<dataset_id>__<region_id>`, this region_id.
2. Added a default setting to select the Bigquery default region.
3. Added `location` attribute to Metadata class.
## Does this introduce a breaking change?
Nope
Co-authored-by: Phani Kumar <94376113+phanikumv@users.noreply.github.com>
Co-authored-by: Felix Uellendall <feluelle@users.noreply.github.com>
Currently, we are assuming all the bigquery datasets to be in the
default region - `US`
If the user tries to access a preexisting table in a different region
say, `europe-west6` all the Temp tables will try to be created in `US`
the region, which will result in the error.
closes: #1369
1. The solution is to create `tmp_astro` region specific. But we cannot
do that since `<project_id>.<dataset_id>` is unique. So we are creating
a unique dataset id like - `<dataset_id>__<region_id>`, this region_id.
2. Added a default setting to select the Bigquery default region.
3. Added `location` attribute to Metadata class.
Nope
Co-authored-by: Phani Kumar <94376113+phanikumv@users.noreply.github.com>
Co-authored-by: Felix Uellendall <feluelle@users.noreply.github.com>
(cherry picked from commit 730d273)
Describe the bug
I am unable to run queries using the
astro.sql.transform
decorator when they do not exist in the region wheretmp_astro
is created. In the below example error, I am running a query on a table inus-central1
, whentmp_astro
exists ineurope-west2
:Version
To Reproduce
Steps to reproduce the behavior:
Expected behavior
I would expect the Astro SDK to make a
tmp_astro
dataset per region, and ensure that it is using/refencing the appropriate tmp schema based on the source table in the query.Additional context
The BigQueryHook supports a location parameter, allowing you to specify the location/region for your query. I don't believe this can be set via a Connection parameter.
The text was updated successfully, but these errors were encountered: