Skip to content

Conversation

@yevgenypats
Copy link
Contributor

@yevgenypats yevgenypats commented Feb 8, 2023

This adds diff methods for Table. This will help simplify migrations so we can transform the destination/database into Table struct and then understand what was changed/added in the same way across all destination plugins.

PostgreSQL PR example - cloudquery/cloudquery#7819

@github-actions github-actions bot added the feat label Feb 8, 2023
@yevgenypats yevgenypats marked this pull request as ready for review February 8, 2023 13:54
@github-actions github-actions bot added feat and removed feat labels Feb 8, 2023
@github-actions
Copy link

github-actions bot commented Feb 8, 2023

⏱️ Benchmark results

  • DefaultConcurrencyDFS-2 resources/s: 11,323
  • DefaultConcurrencyRoundRobin-2 resources/s: 12,107
  • Glob-2 ns/op: 236.4
  • TablesWithChildrenDFS-2 resources/s: 28,382
  • TablesWithChildrenRoundRobin-2 resources/s: 26,615
  • TablesWithRateLimitingDFS-2 resources/s: 28.37
  • TablesWithRateLimitingRoundRobin-2 resources/s: 840.3
  • BufferedScanner-2 ns/op: 11.71
  • LogReader-2 ns/op: 37.19

@codecov-commenter
Copy link

codecov-commenter commented Feb 8, 2023

Codecov Report

Base: 46.73% // Head: 47.25% // Increases project coverage by +0.52% 🎉

Coverage data is based on head (ee722af) compared to base (cb5a47c).
Patch coverage: 56.92% of modified lines in pull request are covered.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #668      +/-   ##
==========================================
+ Coverage   46.73%   47.25%   +0.52%     
==========================================
  Files          69       70       +1     
  Lines        6591     6802     +211     
==========================================
+ Hits         3080     3214     +134     
- Misses       3078     3137      +59     
- Partials      433      451      +18     
Impacted Files Coverage Δ
plugins/source/options.go 0.00% <0.00%> (ø)
schema/column.go 86.66% <ø> (+20.00%) ⬆️
schema/meta.go 10.00% <ø> (ø)
plugins/source/plugin.go 41.14% <40.90%> (-0.07%) ⬇️
schema/table.go 47.11% <79.41%> (+5.74%) ⬆️
internal/memdb/memdb.go 83.57% <100.00%> (ø)
clients/destination/v0/destination_terminate.go 34.61% <0.00%> (-6.57%) ⬇️
plugins/source/testing.go 12.90% <0.00%> (-2.29%) ⬇️
schema/text.go 29.41% <0.00%> (ø)
schema/types.go 68.67% <0.00%> (ø)
... and 6 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

Copy link
Member

@hermanschaaf hermanschaaf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just a few minor suggestions

yevgenypats and others added 3 commits February 8, 2023 17:12
Co-authored-by: Herman Schaaf <hermanschaaf@gmail.com>
Co-authored-by: Herman Schaaf <hermanschaaf@gmail.com>
@kodiakhq kodiakhq bot merged commit f6baa82 into main Feb 8, 2023
@kodiakhq kodiakhq bot deleted the feat/table_diff branch February 8, 2023 15:51
@erezrokah
Copy link
Member

erezrokah commented Feb 8, 2023

kodiakhq bot pushed a commit that referenced this pull request Feb 8, 2023
Good catch, force is already implemented via spec and is the right way to go (I got a bit rusty on the SDK :) ) #668 (comment)
kodiakhq bot pushed a commit that referenced this pull request Feb 8, 2023
🤖 I have created a release *beep* *boop*
---


## [1.36.0](v1.35.0...v1.36.0) (2023-02-08)


### Features

* Add table diff methods ([#668](#668)) ([f6baa82](f6baa82))
* Use Setpgid=true on Unix systems so that signals are not sent to the child process ([#664](#664)) ([2883487](2883487))


### Bug Fixes

* Remove duplicate force implementation ([#670](#670)) ([fe34554](fe34554))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
@yevgenypats
Copy link
Contributor Author

Didn't manage to comment but we already have a migrate_mode: forced in the spec: https://github.com/cloudquery/plugin-sdk/blob/f6baa82d7d1db6d28a47bf3d206306f98aa84bd4/specs/migrate_mode.go

It's used in the SQLite destination with similar logic: https://github.com/cloudquery/cloudquery/blob/4efbf32f77146b61b62956ca9ef41c925d644d4f/plugins/destination/sqlite/client/migrate.go#L242

Thanks , good catch. removed the options thing - #670

kodiakhq bot pushed a commit to cloudquery/cloudquery that referenced this pull request Feb 16, 2023
Blocked by cloudquery/plugin-sdk#668

This streamlines migrations, introduce force option and in general should speed up migrations substantially as we only do now one query instead of query per table. 

Blocked by another take in SDK - cloudquery/plugin-sdk#688
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants