Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(bigquery): add migration quickstart #2249

Merged
merged 7 commits into from Oct 8, 2021

Conversation

shollyman
Copy link
Contributor

No description provided.

@snippet-bot
Copy link

snippet-bot bot commented Sep 28, 2021

Here is the summary of changes.

You are about to add 1 region tag.

This comment is generated by snippet-bot.
If you find problems with this result, please file an issue at:
https://github.com/googleapis/repo-automation-bots/issues.
To update this comment, add snippet-bot:force-run label or use the checkbox below:

  • Refresh this comment

@product-auto-label product-auto-label bot added the api: bigquery Issues related to the BigQuery API. label Sep 28, 2021
@google-cla google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Sep 28, 2021
@product-auto-label product-auto-label bot added the samples Issues that are directly related to samples. label Sep 28, 2021
// Tasks are extensible; the translation task is defined by the BigQuery Migration API.

translationDetails := &translationtaskpb.TranslationTaskDetails{
// TODO: we need to stage some suitable translation inputs and complete this definition.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a GCS path? If so, maybe we could stick something in our sample data bucket?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I've added some resources to the public sample, but I think they still need work.

Tasks: map[string]*migrationpb.MigrationTask{
"example_conversion": {
Type: "Translation_Teradata",
Details: anyDetails,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit confused why the API doesn't use a oneof here. Presumably it only supports a limited number of types + detail messages.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's because the intent is to allow user extension; so you end up with an arbitrary set of task definitions without a central registry.

case <-timeoutCtx.Done():
return nil, fmt.Errorf("task %s didn't complete due to context expiring", workflow.GetName())
default:
polledWorkflow, err := client.GetMigrationWorkflow(timeoutCtx, &migrationpb.GetMigrationWorkflowRequest{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh dear. No long running operation API?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not at alpha at least.

@shollyman
Copy link
Contributor Author

Filed googleapis/repo-automation-bots#2667 to see what's going on with snippet bot

@shollyman
Copy link
Contributor Author

Referencing this here since it's not tracked as part of the PR, but it core to the quickstart:

gsutil cat gs://cloud-samples-data/bigquery/migration/translation/input/queries.txt

DATABASE ExampleDatabase;
SELECT * FROM teratable1;
SELECT invalidcolumn, fname FROM teratable2;
INSERT INTO teratable3 (foo, bar, baz) VALUES (1,2,3);

gsutil cat gs://cloud-samples-data/bigquery/migration/translation/schema/ddl.txt

CREATE TABLE teratable1 (code char, examplejson JSON(1000));
CREATE TABLE teratable2 (idx INT, username VARCHAR(64), fname
VARCHAR(100)) UNIQUE PRIMARY INDEX (idx);

@shollyman
Copy link
Contributor Author

Sample output to GCS:

gsutil cat gs://output prefix omitted/queries.txt

<Translation Failure for query at original line 1, column 1: DATABASE ExampleDatabase;>

-- #Translation Warning# Could not fully verify query, please provide table information for: teratable1
SELECT * FROM teratable1;

-- #Translation Warning# Below Verification Error encountered for translation output. The corresponding input query is at original line 3, column 1.
-- #Translation Warning# Unrecognized name: invalidcolumn [at 1:8]
SELECT invalidcolumn, fname FROM teratable2;

-- #Translation Warning# Could not fully verify query, please provide table information for: teratable3
INSERT INTO teratable3 (foo, bar, baz) VALUES (1, 2, 3);

gsutil cat gs:/output prefix omitted/queries_errors.csv

Input_Query,Line,Column,Category,Message,Type
"DATABASE ExampleDatabase;
",1,1,SqlParseException,Non-query expression encountered in illegal context,ERROR

@shollyman shollyman merged commit 97033f3 into GoogleCloudPlatform:master Oct 8, 2021
@shollyman shollyman deleted the fr-migration branch October 8, 2021 16:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the BigQuery API. cla: yes This human has signed the Contributor License Agreement. samples Issues that are directly related to samples.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants