-
Notifications
You must be signed in to change notification settings - Fork 157
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feature: Partial ("Preview") Syncs #8080
Comments
We do have a |
@edgarrmondragon how hard would it be to make the number of records on the |
@tayloramurphy Well, it uses plugin commands and reports the summary of results if multiple test commands were selected, similar to pytest. Problem is, it's not working at the moment for taps because even after adding a plugins:
extractors:
- name: tap-stackexchange
variant: meltanolabs
pip_url: git+https://github.com/MeltanoLabs/tap-stackexchange.git
commands:
test:
args: --test Meltano doesn't know that it also needs to pass other options, like I've created an issue to address this: #8112. Also, this command can't be used in a pipeline as is because it outputs its test summary information to stdout.
|
Feature scope
API
Description
At Akkio, we have a use case where we want to fetch a limited subset of user-imported datasets. We do this so that we can show users a small preview of what their end data might look like after what's essentially an internal transformation process we run, and we want them to be able to potentially make changes and make sure that transformation process looks good before they commit to the full data sync.
After a brief discussion with @DouweM, this feature doesn't yet seem to exist on Meltano, but is something that would potentially fall under Meltano's purview. The current state of Meltano essentially forces you to do the full data sync before you can actually show it to the user, resulting in subpar experience if the end output isn't really what they want. While split-second queries aren't what Meltano is built for, nor are what I'm really proposing here, allowing for a preview window of sorts still seems like a positive addition cohesive with the rest of the tool to me.
Dropping some other comments that were made, for full context:
The text was updated successfully, but these errors were encountered: