Skip to content

Conversation

@hermanschaaf
Copy link
Member

This adds a backwards-compatible Validation function to the SDK. It's backwards-compatible in the sense that even if it is called from a new CLI using old plugin versions that don't support it, it still works. In this case, a warning is returned that the plugin should be updated.

Combined with a few straight-forward changes in the CLI, we can provide users with warnings (such as deprecation warnings) and short, descriptive errors that are easier to read.

Some examples of using this in the CLI:

../cloudquery/cli/cli sync aws.yml
Loading spec(s) from aws.yml
Error: (aws) name aws_ec2_reserved_instances does not match any known table names
Error: failed to sync source aws: validation for source plugin aws failed
../cloudquery/cli/cli sync aws.yml
Loading spec(s) from aws.yml
Warning: (aws) "concurrency" is deprecated and will be removed in a future version: use "table_concurrency" and "resource_concurrency" instead.
Warning: (postgresql) the version of this plugin is outdated and should be updated
Starting sync for:  aws -> [postgresql]
../cloudquery/cli/cli sync aws.yml
Loading spec(s) from aws.yml
Warning: (postgresql) the version of this plugin is outdated and should be updated
Starting sync for:  aws -> [postgresql]
Sync completed successfully.
Summary: resources: 3856, errors: 1, panic: 0 failed_writes: 0

Opening this as a Draft for some early feedback. If you are happy with the idea, I can tie up some loose ends and add tests.

@hermanschaaf hermanschaaf requested a review from disq October 10, 2022 15:44
return tables, nil
}

func (c *SourceClient) Validate(ctx context.Context, spec specs.Source) (warnings, errors []string, err error) {
Copy link
Member

@disq disq Oct 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apart from the errors shadow/nit, could maybe consider returning a ValidateResult struct instead of the triple return values? But then it would be very similar to the grpc structs, ValidateDestination_Response etc. Maybe there's a better way? Pass a logger (as interface) and let it log? It's a bunch of lines, wouldn't need any formatting on the CLI side anyway.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I'd be good with a struct; it's more extensible in the long run too. I don't think it's an issue that it's similar to ValidateDestination_Response.

if ok && st.Code() == codes.Unimplemented {
// Backwards-compatibility with older plugin versions that don't support Validate().
// In this case, we only return one warning: that the plugin should be updated.
return []string{"the version of this plugin is outdated and should be updated"}, nil, nil

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Maybe temper the wording a bit.
Some advanced features might not be supported by the current version of the CLI, and might lead to breaks in future. Please update using (this link)

Copy link
Member Author

@hermanschaaf hermanschaaf Oct 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, good suggestion - I was struggling to find the right wording here :)

hermanschaaf added a commit that referenced this pull request Oct 11, 2022
See cloudquery/cloudquery#2612 for an
explanation.

Notes:
- We weren't using `DisallowUnknownFields` on the destination plugin
side
- We don't currently have a mechanism to warn about unknown fields, but
we are working on it #272
- This will unfortunately not fix plugins that have already been
released. Our next CLI release that adds new fields will have to be
breaking, but after that they can be backwards-compatible.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants