feat: Arrow migration for destination #737
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportPatch coverage:
Additional details and impacted files@@ Coverage Diff @@
## main #737 +/- ##
==========================================
+ Coverage 43.21% 46.86% +3.65%
==========================================
Files 80 76 -4
Lines 7889 7788 -101
==========================================
+ Hits 3409 3650 +241
+ Misses 4013 3647 -366
- Partials 467 491 +24
... and 9 files with indirect coverage changes Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. ☔ View full report in Codecov by Sentry. |
erezrokah
reviewed
Mar 26, 2023
Member
erezrokah
left a comment
erezrokah
left a comment
There was a problem hiding this comment.
As an initial PR this looks good, I'll do a proper review once it ready
yevgenypats
force-pushed
the
feat/arrow_update
branch
from
fa2a2de to
65899e0
Compare
hermanschaaf
reviewed
Apr 3, 2023
hermanschaaf
reviewed
Apr 3, 2023
hermanschaaf
reviewed
Apr 3, 2023
hermanschaaf
reviewed
Apr 3, 2023
hermanschaaf
reviewed
Apr 3, 2023
hermanschaaf
reviewed
Apr 3, 2023
Co-authored-by: Herman Schaaf <hermanschaaf@gmail.com>
hermanschaaf
reviewed
Apr 3, 2023
erezrokah
approved these changes
Apr 10, 2023
Member
erezrokah
left a comment
erezrokah
left a comment
There was a problem hiding this comment.
This is quite a bit of code, but I think I get the general idea of replacing our type system with Arrow's + some extensions.
Might be better to merge and work of main so we can test thoroughly.
Had some comment, the most important (non blocking one) is about the internal columns
Member
|
We also need to fix the CI and tests |
Co-authored-by: Erez Rokah <erezrokah@users.noreply.github.com>
This is a follow-up on the latest commit because we were adding just the columns without the data to the records so this should solve it by moving it up to the server
⏱️ Benchmark resultsComparing with 336a957
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is ready for first review with tests passing. might need some more manual testing and maybe migrating a plugin like mongo/postgres and testing with source like aws,gcp to stress test common use-cases and not only internal memdb
BEGIN_COMMIT_OVERRIDE
feat!: Arrow migration for destination
END_COMMIT_OVERRIDE