Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Stronger separation of ingestion phase #155

Closed
ericbusboom opened this issue Nov 27, 2015 · 0 comments
Closed

Stronger separation of ingestion phase #155

ericbusboom opened this issue Nov 27, 2015 · 0 comments
Assignees
@ericbusboom
Labels
Improvement
Milestone
Ambry 1.0

Comments

@ericbusboom
Copy link
Contributor

Create a stronger separation of the ingestion phase.

Don't auto create source tables. Create them only when the wranger requests it. The source tables are always loaded in from config.

So, the wrangler runs an ingest, then exports the source tables, and edits them. Then they are synced in.

When a source is run, if a source table exists, a new pipe segment is created to map the source headers and column types.

This change would also make it easier to load sources directly into build pipelines.
@ericbusboom ericbusboom self-assigned this Nov 27, 2015
@ericbusboom ericbusboom added this to the Ambry 1.0 milestone Nov 27, 2015
@ericbusboom ericbusboom reopened this Nov 27, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ericbusboom ericbusboom

Labels
Improvement
Projects
None yet
Milestone
Ambry 1.0
Development

No branches or pull requests
1 participant
@ericbusboom