Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add normalization #75

Closed
revit13 opened this issue Jan 31, 2023 · 0 comments · Fixed by #76
Closed

Add normalization #75

revit13 opened this issue Jan 31, 2023 · 0 comments · Fixed by #76
Assignees
@cdoron @revit13

Comments

@revit13
Copy link
Contributor

revit13 commented Jan 31, 2023

Without normalization, the table are in airbyte format. Thus normalization process should be done to transform the data to the expected form.

Reference: https://docs.airbyte.com/understanding-airbyte/basic-normalization/
revit13 added a commit to revit13/airbyte-module that referenced this issue Feb 2, 2023
@revit13
Add normalization (fybrik#75)
23d2de7 
Signed-off-by: Revital Sur <eres@il.ibm.com>
@revit13 revit13 mentioned this issue Feb 2, 2023
Merged
revit13 added a commit to revit13/airbyte-module that referenced this issue Feb 2, 2023
@revit13 @cdoron
Add normalization (fybrik#75)
c3b4a54 
Signed-off-by: Revital Sur <eres@il.ibm.com>
Co-authored-by: Doron Chen <cdoron@il.ibm.com>
revit13 added a commit to revit13/airbyte-module that referenced this issue Feb 4, 2023
@revit13 @cdoron
Add normalization (fybrik#75)
9b05db6 
Signed-off-by: Revital Sur <eres@il.ibm.com>
Co-authored-by: Doron Chen <cdoron@il.ibm.com>
revit13 added a commit to revit13/airbyte-module that referenced this issue Feb 5, 2023
@revit13 @cdoron
Add normalization (fybrik#75)
657c62c 
Signed-off-by: Revital Sur <eres@il.ibm.com>
Co-authored-by: Doron Chen <cdoron@il.ibm.com>
cdoron added a commit that referenced this issue Feb 9, 2023
@revit13 @cdoron
Add normalization (#76)
36f0155 
* Add normalization (#75)

Signed-off-by: Revital Sur <eres@il.ibm.com>
Co-authored-by: Doron Chen <cdoron@il.ibm.com>

* Change the run command of the normalization image.

Signed-off-by: Revital Sur <eres@il.ibm.com>

* Add container.py for common code to handle container functionality.

Signed-off-by: Revital Sur <eres@il.ibm.com>

* Apply normalization only on write operation.

Signed-off-by: Revital Sur <eres@il.ibm.com>

* Add a table to the asset.

Signed-off-by: Revital Sur <eres@il.ibm.com>

* Fixes after testing.

Signed-off-by: Revital Sur <eres@il.ibm.com>

* Add overwrite mode for write operation.

Signed-off-by: Revital Sur <eres@il.ibm.com>

* Minor fix.

Signed-off-by: Revital Sur <eres@il.ibm.com>

* Address review comments.

Signed-off-by: Revital Sur <eres@il.ibm.com>

* Remove the usage of transformer.transform in extract_data.

Signed-off-by: Revital Sur <eres@il.ibm.com>

* Address Doron's comments.

Signed-off-by: Revital Sur <eres@il.ibm.com>

* Minor changes.

Signed-off-by: Revital Sur <eres@il.ibm.com>

* Add comments.

Signed-off-by: Revital Sur <eres@il.ibm.com>

* Rename function name in abm/connector.py

Signed-off-by: Revital Sur <eres@il.ibm.com>

* Minor fix.

Signed-off-by: Revital Sur <eres@il.ibm.com>

* Minor change.

Signed-off-by: Revital Sur <eres@il.ibm.com>

* Address review comments.

Signed-off-by: Revital Sur <eres@il.ibm.com>

---------

Signed-off-by: Revital Sur <eres@il.ibm.com>
Co-authored-by: Doron Chen <cdoron@il.ibm.com>
@cdoron cdoron closed this as completed in #76 Feb 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cdoron cdoron

@revit13 revit13

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add normalization revit13/airbyte-module
2 participants
@cdoron @revit13