Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Couldn't use caching when the connector is BigQuery #261

Closed
cyyeh opened this issue Jul 31, 2023 · 0 comments · Fixed by #268
Closed

Couldn't use caching when the connector is BigQuery #261

cyyeh opened this issue Jul 31, 2023 · 0 comments · Fixed by #268
Assignees
Labels
needs-triage type/bug Something went wrong

Comments

@cyyeh
Copy link
Member

cyyeh commented Jul 31, 2023

Environment

  • Version:

    cli version: 0.7.1
    core version: 0.7.1
    build version: 0.7.1
    serve version: 0.7.1
    
  • OS: macOS with M1 chip

What's wrong?

Couldn't use caching when connector is BigQuery. After discussing with @onlyjackfrost, there might be two places we need to modify in packages/extension-driver-bq/src/lib/bgDataSource.ts:

// Use the directory to avoid filename collision,
    // The wildcard indicates the partition,
    // ref: https://cloud.google.com/bigquery/docs/reference/standard-sql/other-statements#export_option_list
    const bucketObjPrefix = path.join(bucketName, directory);
    const uri = `gs://${path.join(bucketObjPrefix, 'part*.parquet')}`;
    const queryOptions = {
      query: `EXPORT DATA OPTIONS( uri="${uri}", format='PARQUET') AS ${statement}`,
      location: options?.location || 'US',
    };
const getFilesResponse = await storage.bucket(bucketName).getFiles({
  prefix: directory.slice(1),
})

What's the correct behavior?

Users can caching when the connector is BigQuery

Reproducing steps

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs-triage type/bug Something went wrong
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants