Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(warehouse): staging file schema consolidation #3088

Merged
merged 95 commits into from
May 22, 2023

Conversation

achettyiitr
Copy link
Member

@achettyiitr achettyiitr commented Mar 9, 2023

Description

  • Consolidation of staging files schema with the warehouse schema
    • Before
      • For consolidating staging files, we iterate over all the staging files and consolidate them with warehouse schema.
    • After
      • Now we consolidate all the staging files first.
      • Later it is then consolidated with the warehouse schema.
  • Refactoring around the schema package.

Notion Ticket

https://www.notion.so/rudderstacks/Consolidation-staging-files-schema-846379613cae451dbb5154d5903310cf?pvs=4

Security

  • The code changed/added as part of this pull request won't create any security issues with how the software is being used.

@achettyiitr achettyiitr changed the base branch from master to feat.schema-repository March 9, 2023 17:38
Copy link
Collaborator

@fracasula fracasula left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is returning an error for me, can you please check it out? I think the fetchSchemaFromWarehouse method takes the fetchSchemaRepo interface:

type fetchSchemaRepo interface {
	FetchSchema(warehouse model.Warehouse) (model.Schema, model.Schema, error)
}

Which as the warehouse parameter as the only parameter. The problem is that whManager is of type Manager which has:

type Manager interface {
	FetchSchema() (model.Schema, model.Schema, error)

No parameter there.

warehouse/schema.go Outdated Show resolved Hide resolved
@achettyiitr achettyiitr merged commit a616e8b into master May 22, 2023
25 checks passed
@achettyiitr achettyiitr deleted the feat.schema-consolidation branch May 22, 2023 08:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants