🎉 Add docs on custom Operations #3507

ChristopheDuong · 2021-05-20T16:59:59Z

What

┆Issue is synchronized with this Asana task by Unito

docs/tutorials/transformation-and-normalization/transformations-with-dbt.md

davinchia

Thanks for this @ChristopheDuong ! This is awesome.

Most of my comments are phrasing suggestions. Two medium things about better organising the operations page and figuring out a good place to place the steps on dbt with private repos.

I know I commented a bunch about UI snapshot - I realise this is blocking on Artem. Either removing those lines or waiting for him to merge is fine by me.

Not approving until we figure those out.

davinchia · 2021-05-21T03:19:17Z

docs/tutorials/transformation-and-normalization/transformations-with-dbt.md

@@ -52,21 +59,34 @@ Connection:
  keepalives_idle: 0
  sslmode: None
  Connection test: OK connection ok
+```


suggestion: small comment pointing out what to look for. it's a lot to parse otherwise.

my comment is less relevant if 'everything' being 'ok' is what we expect

Yes, everything should be 'OK'

It's mainly to see the expected output of the dbt command look like too

davinchia · 2021-05-21T03:25:37Z

docs/tutorials/transformation-and-normalization/transformations-with-dbt.md

-select * from covid_epidemiology_with_id
+    ]) }} as _airbyte_covid_epidemiology_hashid
+from {{ ref('covid_epidemiology_ab2_558') }}
+-- covid_epidemiology


love this example!

davinchia · 2021-05-21T03:28:23Z

docs/understanding-airbyte/basic-normalization.md

@@ -248,3 +248,6 @@ Note that all the choices made by Normalization as described in this documentati
 * to build a [custom SQL view]() with your own naming conventions
 * to export, edit and run [custom DBT normalization](https://github.com/airbytehq/airbyte/tree/e378d40236b6a34e1c1cb481c8952735ec687d88/docs/tutorials/transformation-and-normalization/transformations-with-dbt.md) yourself

+## UI Configurations
+
+Screenshots of examples of normalization settings here


is this WIP?

should we remove before merging in?

docs/tutorials/transformation-and-normalization/transformations-with-dbt.md

davinchia · 2021-05-21T03:32:14Z

docs/tutorials/transformation-and-normalization/transformations-with-dbt.md

+
+    run --models tag:covid_api opendata.base.*
+
+Screenshots of examples of normalization settings for my private git repo here


same comment on making sure we do this before merging.

docs/tutorials/transformation-and-normalization/transformations-with-sql.md

docs/understanding-airbyte/operations.md

Co-authored-by: Davin Chia <davinchia@gmail.com>

ChristopheDuong · 2021-05-21T16:38:37Z

I'm marking the PR it as draft while waiting for UI implementation to add the screenshots

avaidyanatha

This may seem like a lot of comments, but it's mostly just wording/grammar/clarity fixes - thank you so much @ChristopheDuong for writing this up :) ❤️

EDIT: Also I realize that this PR is blocked on a few other things as @davinchia mentioned. So take my approval as more of a check on wording/clarity rather than an OK to merge 😅

docs/tutorials/transformation-and-normalization/transformations-with-airbyte.md

avaidyanatha · 2021-05-21T19:58:59Z

docs/tutorials/transformation-and-normalization/transformations-with-airbyte.md

+After replication of data from a source connector (Extract) to a destination connector (Load), multiple optional transformations steps can now be applied as part of an Airbyte Sync. Possible transformations are:
+
+1. Basic normalization transformations as automatically generated by Airbyte dbt code generator.
+2. Customized normalization transformations as edited by the user (and disable the default generated normalization one)


Suggested change

2. Customized normalization transformations as edited by the user (and disable the default generated normalization one)

2. Customized normalization transformations as edited by the user (which disables the default generated normalization one)

docs/tutorials/transformation-and-normalization/transformations-with-airbyte.md

docs/understanding-airbyte/operations.md

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

davinchia

Feel free to merge whenever you add all the pictures!

marcosmarxm · 2021-06-01T14:20:23Z

docs/understanding-airbyte/connections/README.md

+* [Full Refresh Overwrite](full-refresh-overwrite.md): Sync the whole stream and replace data in destination by overwriting it.
+* [Full Refresh Append](full-refresh-append.md): Sync the whole stream and append data in destination.
+* [Incremental Append](incremental-append.md): Sync new records from stream and append data in destination.
+* [Incremental Deduped History](incremental-deduped-history.md): Sync new records from stream and append data in destination, also provides a de-duplicated view mirroring the state of the stream in the source.


@ChristopheDuong can add a note saying that for the first run Incremental mode works as a full refresh ?

Add docs on custom Operations

5f9161f

auto-assign bot requested review from cgardens and davinchia May 20, 2021 17:00

ChristopheDuong added 2 commits May 20, 2021 19:00

fix test

4c23547

Tweak docs

53c9662

davinchia reviewed May 21, 2021

View reviewed changes

docs/tutorials/transformation-and-normalization/transformations-with-dbt.md Outdated Show resolved Hide resolved

davinchia reviewed May 21, 2021

View reviewed changes

Apply suggestions from code review

9acb46c

Co-authored-by: Davin Chia <davinchia@gmail.com>

ChristopheDuong requested a review from avaidyanatha May 21, 2021 08:54

ChristopheDuong added 3 commits May 21, 2021 13:56

Tweak docs

7e347ef

Merge remote-tracking branch 'origin/master' into custom-dbt-docs

9d71608

Change tutorials links

127349c

ChristopheDuong marked this pull request as draft May 21, 2021 16:38

avaidyanatha approved these changes May 21, 2021

View reviewed changes

ChristopheDuong and others added 3 commits May 25, 2021 11:26

Apply suggestions from code review

ed6ee98

Co-authored-by: Abhi Vaidyanatha <abhi@airbyte.io>

Reword

384b956

Add screenshots to docs

1ded71c

davinchia approved these changes Jun 1, 2021

View reviewed changes

Add note on first sync with incremental

198762d

marcosmarxm reviewed Jun 1, 2021

View reviewed changes

Merge remote-tracking branch 'origin/master' into custom-dbt-docs

97f8f38

ChristopheDuong changed the base branch from master to chris/namespace-docs June 4, 2021 12:39

ChristopheDuong changed the base branch from chris/namespace-docs to master June 4, 2021 12:40

avaidyanatha mentioned this pull request Jun 8, 2021

[TOPIC] Airbyte's approach to transformation airbytehq/write-for-the-community#25

Closed

ChristopheDuong added 2 commits June 11, 2021 15:07

Merge remote-tracking branch 'origin/master' into custom-dbt-docs

31fdb1a

Add example for custom transformation with public git

5491b44

github-actions bot added the area/documentation Improvements or additions to documentation label Jun 11, 2021

ChristopheDuong marked this pull request as ready for review June 14, 2021 08:31

ChristopheDuong merged commit 22b75c5 into master Jun 14, 2021

ChristopheDuong deleted the chris/custom-dbt-docs branch June 14, 2021 08:31

ChristopheDuong changed the title ~~Add docs on custom Operations~~ 🎉 Add docs on custom Operations Jun 14, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Add docs on custom Operations #3507

🎉 Add docs on custom Operations #3507

ChristopheDuong commented May 20, 2021 •

edited by sync-by-unito bot

Loading

davinchia left a comment

davinchia May 21, 2021

ChristopheDuong May 21, 2021

davinchia May 21, 2021

davinchia May 21, 2021

davinchia May 21, 2021

davinchia May 21, 2021

ChristopheDuong commented May 21, 2021 •

edited

Loading

avaidyanatha left a comment •

edited

Loading

avaidyanatha May 21, 2021

davinchia left a comment

marcosmarxm Jun 1, 2021


		run --models tag:covid_api opendata.base.*

		Screenshots of examples of normalization settings for my private git repo here

	2. Customized normalization transformations as edited by the user (and disable the default generated normalization one)
	2. Customized normalization transformations as edited by the user (which disables the default generated normalization one)

🎉 Add docs on custom Operations #3507

🎉 Add docs on custom Operations #3507

Conversation

ChristopheDuong commented May 20, 2021 • edited by sync-by-unito bot Loading

What

davinchia left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ChristopheDuong commented May 21, 2021 • edited Loading

avaidyanatha left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davinchia left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ChristopheDuong commented May 20, 2021 •

edited by sync-by-unito bot

Loading

ChristopheDuong commented May 21, 2021 •

edited

Loading

avaidyanatha left a comment •

edited

Loading