From 6784670653fd6ce1ad2b0405686e83c290c29b55 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Patricija=20Bre=C4=8Dko?= Date: Mon, 24 Mar 2025 12:49:24 +0100 Subject: [PATCH 01/27] feat: revamp airdrop docs --- .../connecting-to-the-external-system.mdx | 18 ++++ fern/docs/pages/airdrop/development-guide.mdx | 7 ++ fern/docs/pages/airdrop/getting-started.mdx | 85 ++++++++++++------- fern/docs/pages/airdrop/how-does-it-work.mdx | 5 ++ fern/docs/pages/airdrop/overview.mdx | 19 +---- .../pages/airdrop/publish-to-marketplace.mdx | 3 + fern/docs/pages/airdrop/snap-in-examples.mdx | 5 ++ fern/versions/public.yml | 31 +++++-- 8 files changed, 117 insertions(+), 56 deletions(-) create mode 100644 fern/docs/pages/airdrop/connecting-to-the-external-system.mdx create mode 100644 fern/docs/pages/airdrop/development-guide.mdx create mode 100644 fern/docs/pages/airdrop/how-does-it-work.mdx create mode 100644 fern/docs/pages/airdrop/publish-to-marketplace.mdx create mode 100644 fern/docs/pages/airdrop/snap-in-examples.mdx diff --git a/fern/docs/pages/airdrop/connecting-to-the-external-system.mdx b/fern/docs/pages/airdrop/connecting-to-the-external-system.mdx new file mode 100644 index 00000000..21b98f71 --- /dev/null +++ b/fern/docs/pages/airdrop/connecting-to-the-external-system.mdx @@ -0,0 +1,18 @@ +TODO: Keyring explanation and guide setting up. + +## Keyrings + +_Keyrings_ are a DevRev-specific mechanism for managing authentication to external systems. +They are called **Connections** in the DevRev app. + +Keyrings provide a secure way to store and manage credentials within your DevRev snap-in. +This eliminates the need to expose sensitive information like passwords or access tokens directly +within your code or configuration files, enhancing overall security. +They also provide a valid token by abstracting OAuth token renewal from the end user. + +A keyring is used by a worker to authenticate to the external system in API calls. +They include the key (for example, a PAT or API key), its type, the organization ID for which a +key is valid, and in some cases the organization name. + +Configure a keyring for the external system in the `manifest.yaml`. +Refer to [Keyrings](/snapin-development/references/manifest#developer-keyrings) for instructions. diff --git a/fern/docs/pages/airdrop/development-guide.mdx b/fern/docs/pages/airdrop/development-guide.mdx new file mode 100644 index 00000000..7104b493 --- /dev/null +++ b/fern/docs/pages/airdrop/development-guide.mdx @@ -0,0 +1,7 @@ +## Extracting data from external system + +TODO: Explain the order of development steps. + +## Loading data to DevRev + +TODO: Explain the order of development steps. diff --git a/fern/docs/pages/airdrop/getting-started.mdx b/fern/docs/pages/airdrop/getting-started.mdx index c0023364..49e1d269 100644 --- a/fern/docs/pages/airdrop/getting-started.mdx +++ b/fern/docs/pages/airdrop/getting-started.mdx @@ -1,35 +1,54 @@ -## Before you begin - -- If you have not created a development organization on DevRev before, create a dedicated -[DevRev organization for development purposes](https://app.devrev.ai/) where you are -publishing your Airdrop snap-in. -- Install required tools and packages: - - [devrev-cli](https://developer.devrev.ai/snapin-development/references/cli-install) (version 4.7 or higher) - - [jq](https://stedolan.github.io/jq) - - [golang](https://go.dev/doc/install) - - [nodejs](https://nodejs.org/en/download/package-manager) (version 18.x.x+ or higher) - -## Setting up the snap-in - -1. Create a new repo from the [Airdrop template repo](https://github.com/devrev/adaas-template). -Select _Use this template_ and then _Create a new repository_. -1. Copy `.env.example` to `.env` and fill in the required variables. -1. Configure a keyring for the external system in the `manifest.yaml`. - Refer to [Keyrings](/snapin-development/references/manifest#developer-keyrings) for instructions. -1. Deploy a draft version of your snap-in to your dev org using `make deploy`. -1. Install the snap-in in your DevRev at **Settings** > **Snap-ins** > **Install snap-in**. -1. Define the connection at **Settings** > **Imports** > **Connections**. -1. Create an import at **Settings** > **Imports** > **Import**. - -## Observability - -- To observe logs from your snap-in in your development environment: - ``` - devrev snap_in_package logs | jq - ``` -- To open logs in your favorite editor: - ``` - devrev snap_in_package logs | code - - ``` +## Before You Begin + +If this is your first time setting up an organization within DevRev, you'll need to create a dedicated DevRev organization for developing your new Airdrop snap-in: + +1. Click on your profile picture in the top left corner. +2. Navigate to the **Orgs** section and click on **+**. +3. Fill in the necessary details and click **Create**. + +You will now find your newly created organization under **Orgs**. + +Install required tools and packages: + +- [devrev-cli](https://developer.devrev.ai/snapin-development/references/cli-install) (version 4.7 or higher) +- [jq](https://stedolan.github.io/jq) +- [golang](https://go.dev/doc/install) +- [nodejs](https://nodejs.org/en/download/package-manager) (version 18.x.x+ or higher) + +## Setting Up the Snap-in + +### Starter Template + +DevRev provides a starter template for Airdrop snap-in development. + +1. Create a new repository from [Airdrop template repository](https://github.com/devrev/adaas-template) by selecting _Use this template_ and then _Create a new repository_. +1. Copy `.env.example` to a new file named `.env` and fill in the required variables. + +### Local Development + +For ease of development, you can run your snap-in locally and receive logs directly in your console. Follow the guide [here](/snapin-development/locally-testing-snap-ins). + +### Publishing the snap-in + +Once you're ready to launch your snap-in in a production environment, follow these steps: + +1. Deploy a draft version of your snap-in to your organization by using `make deploy`. +1. Install the snap-in in your DevRev by going to **Settings** > **Snap-ins** > **Install snap-in**. +1. Set up the connection under **Settings** > **Airdrops** > **Connections**. +1. Create an import at **Settings** > **Airdrops** > **Airdrop**. + +#### Observability + +To observe logs from your snap-in in your development environment: + +``` +devrev snap_in_package logs | jq +``` + +To open logs in your favorite editor: + +``` +devrev snap_in_package logs | code - +``` For more information, refer to [Debugging](/snapin-development/debugging). diff --git a/fern/docs/pages/airdrop/how-does-it-work.mdx b/fern/docs/pages/airdrop/how-does-it-work.mdx new file mode 100644 index 00000000..109b179e --- /dev/null +++ b/fern/docs/pages/airdrop/how-does-it-work.mdx @@ -0,0 +1,5 @@ +TODO: Explain how ADaaS code works by using the template + +TODO: Explain the role of manifest.yaml and where is what + +TODO: Explain how the communication between snap-in and DevRev works (through emitting events). diff --git a/fern/docs/pages/airdrop/overview.mdx b/fern/docs/pages/airdrop/overview.mdx index a74a1199..cdc4eaea 100644 --- a/fern/docs/pages/airdrop/overview.mdx +++ b/fern/docs/pages/airdrop/overview.mdx @@ -8,9 +8,6 @@ DevRev’s Airdrop functionality. It enables developers to create external workers (extractors and loaders) to bring data from various external systems. -DevRev offers a repository template to help you create Airdrop snap-ins. -Refer to [Getting started](getting-started) for instructions. - ```mermaid flowchart TD %%{init: { @@ -57,18 +54,4 @@ An extractor function in the snap-in is responsible for extracting data from the A _reverse sync_ is a sync run from DevRev to an external system. It uses a loader function, to create or update data in the external system. -## Keyrings - -_Keyrings_ are a DevRev-specific mechanism for managing authentication to external systems. -They are called **Connections** in the DevRev app. - -Keyrings provide a secure way to store and manage credentials within your DevRev snap-in. -This eliminates the need to expose sensitive information like passwords or access tokens directly -within your code or configuration files, enhancing overall security. -They also provide a valid token by abstracting OAuth token renewal from the end user. - -A keyring is used by a worker to authenticate to the external system in API calls. -They include the key (for example, a PAT or API key), its type, the organization ID for which a -key is valid, and in some cases the organization name. - -Refer to [Keyrings](/snapin-development/references/keyrings) for more information. +TODO: 1-way sync, 2-way sync, incremental sync, ... diff --git a/fern/docs/pages/airdrop/publish-to-marketplace.mdx b/fern/docs/pages/airdrop/publish-to-marketplace.mdx new file mode 100644 index 00000000..76f90301 --- /dev/null +++ b/fern/docs/pages/airdrop/publish-to-marketplace.mdx @@ -0,0 +1,3 @@ +When you feel like your snap-in is ready to be used by different organizations, you can publish it to the DevRev marketplace. + +Follow this [guide](/snapin-development/marketplace-listings). diff --git a/fern/docs/pages/airdrop/snap-in-examples.mdx b/fern/docs/pages/airdrop/snap-in-examples.mdx new file mode 100644 index 00000000..e5221556 --- /dev/null +++ b/fern/docs/pages/airdrop/snap-in-examples.mdx @@ -0,0 +1,5 @@ +You can find snap-in examples for various external systems [here](https://github.com/devrev/adaas-connector-examples). + +Examples include: + +- Asana diff --git a/fern/versions/public.yml b/fern/versions/public.yml index 7886f45c..f700b6b1 100644 --- a/fern/versions/public.yml +++ b/fern/versions/public.yml @@ -145,6 +145,24 @@ navigation: - page: "Getting started" slug: getting-started path: ../docs/pages/airdrop/getting-started.mdx + - page: "How does it work" + slug: how-does-it-work + path: ../docs/pages/airdrop/how-does-it-work.mdx + - page: "Snap-in examples" + slug: snap-in-examples + path: ../docs/pages/airdrop/snap-in-examples.mdx + - page: "Connecting to the external system" + slug: connecting-to-the-external-system + path: ../docs/pages/airdrop/connecting-to-the-external-system.mdx + - section: "Development guide" + slug: development-guide + contents: + - page: "Extracting data from external system" + slug: extracting-data + path: ../docs/pages/airdrop/development-guide/extracting-data.mdx + - page: "Loading data to DevRev" + slug: loading-data + path: ../docs/pages/airdrop/development-guide/loading-data.mdx - page: "Extraction phases" slug: extraction-phases path: ../docs/pages/airdrop/extraction-phases.mdx @@ -154,24 +172,27 @@ navigation: path: ../docs/pages/airdrop/loading-phases.mdx - page: "External sync units extraction" slug: external-sync-units-extraction - hidden: false + hidden: true path: ../docs/pages/airdrop/external-sync-units-extraction.mdx - page: "Metadata extraction" - hidden: false + hidden: true slug: metadata-extraction path: ../docs/pages/airdrop/metadata-extraction.mdx - page: "Data extraction" - hidden: false + hidden: true slug: data-extraction path: ../docs/pages/airdrop/data-extraction.mdx - page: "Attachments extraction" - hidden: false + hidden: true slug: extract-attachments path: ../docs/pages/airdrop/attachments-extraction.mdx - page: "Data and attachments deletion" slug: data-attachments-deletion - hidden: false + hidden: true path: ../docs/pages/airdrop/data-attachments-deletion.mdx + - page: "Publish to the marketplace" + slug: publish-to-marketplace + path: ../docs/pages/airdrop/publish-to-marketplace.mdx - page: "Common issues - FAQ" slug: faq path: ../docs/pages/airdrop/faq.mdx From ad88f55816fc5670e6b83a6580cdbb50fbb5dcfc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Patricija=20Bre=C4=8Dko?= Date: Mon, 24 Mar 2025 12:56:55 +0100 Subject: [PATCH 02/27] Add missing pages --- fern/docs/pages/airdrop/extracting-data.mdx | 1 + fern/docs/pages/airdrop/loading-data.mdx | 1 + fern/versions/public.yml | 4 ++-- 3 files changed, 4 insertions(+), 2 deletions(-) create mode 100644 fern/docs/pages/airdrop/extracting-data.mdx create mode 100644 fern/docs/pages/airdrop/loading-data.mdx diff --git a/fern/docs/pages/airdrop/extracting-data.mdx b/fern/docs/pages/airdrop/extracting-data.mdx new file mode 100644 index 00000000..1333ed77 --- /dev/null +++ b/fern/docs/pages/airdrop/extracting-data.mdx @@ -0,0 +1 @@ +TODO diff --git a/fern/docs/pages/airdrop/loading-data.mdx b/fern/docs/pages/airdrop/loading-data.mdx new file mode 100644 index 00000000..1333ed77 --- /dev/null +++ b/fern/docs/pages/airdrop/loading-data.mdx @@ -0,0 +1 @@ +TODO diff --git a/fern/versions/public.yml b/fern/versions/public.yml index f700b6b1..6c1572ab 100644 --- a/fern/versions/public.yml +++ b/fern/versions/public.yml @@ -159,10 +159,10 @@ navigation: contents: - page: "Extracting data from external system" slug: extracting-data - path: ../docs/pages/airdrop/development-guide/extracting-data.mdx + path: ../docs/pages/airdrop/extracting-data.mdx - page: "Loading data to DevRev" slug: loading-data - path: ../docs/pages/airdrop/development-guide/loading-data.mdx + path: ../docs/pages/airdrop/loading-data.mdx - page: "Extraction phases" slug: extraction-phases path: ../docs/pages/airdrop/extraction-phases.mdx From f12d500b9592895ad5129edaa8782d6ef8c7ae3b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Patricija=20Bre=C4=8Dko?= Date: Mon, 24 Mar 2025 13:08:45 +0100 Subject: [PATCH 03/27] Update getting started --- fern/docs/pages/airdrop/getting-started.mdx | 25 ++++++++++++--------- 1 file changed, 14 insertions(+), 11 deletions(-) diff --git a/fern/docs/pages/airdrop/getting-started.mdx b/fern/docs/pages/airdrop/getting-started.mdx index 49e1d269..95aaff6c 100644 --- a/fern/docs/pages/airdrop/getting-started.mdx +++ b/fern/docs/pages/airdrop/getting-started.mdx @@ -1,30 +1,33 @@ -## Before You Begin +## Before you begin + +### Create a new organization If this is your first time setting up an organization within DevRev, you'll need to create a dedicated DevRev organization for developing your new Airdrop snap-in: 1. Click on your profile picture in the top left corner. -2. Navigate to the **Orgs** section and click on **+**. +2. Go to the **Orgs** section and click on **+**. 3. Fill in the necessary details and click **Create**. -You will now find your newly created organization under **Orgs**. +You now find your newly created organization under **Orgs**. -Install required tools and packages: +### Install required tools and packages - [devrev-cli](https://developer.devrev.ai/snapin-development/references/cli-install) (version 4.7 or higher) - [jq](https://stedolan.github.io/jq) - [golang](https://go.dev/doc/install) - [nodejs](https://nodejs.org/en/download/package-manager) (version 18.x.x+ or higher) +- [ngrok](https://ngrok.com) -## Setting Up the Snap-in +## Setting up the snap-in -### Starter Template +### Starter template DevRev provides a starter template for Airdrop snap-in development. 1. Create a new repository from [Airdrop template repository](https://github.com/devrev/adaas-template) by selecting _Use this template_ and then _Create a new repository_. -1. Copy `.env.example` to a new file named `.env` and fill in the required variables. +2. Copy `.env.example` to a new file named `.env` and fill in the required variables. -### Local Development +### Local development For ease of development, you can run your snap-in locally and receive logs directly in your console. Follow the guide [here](/snapin-development/locally-testing-snap-ins). @@ -33,9 +36,9 @@ For ease of development, you can run your snap-in locally and receive logs direc Once you're ready to launch your snap-in in a production environment, follow these steps: 1. Deploy a draft version of your snap-in to your organization by using `make deploy`. -1. Install the snap-in in your DevRev by going to **Settings** > **Snap-ins** > **Install snap-in**. -1. Set up the connection under **Settings** > **Airdrops** > **Connections**. -1. Create an import at **Settings** > **Airdrops** > **Airdrop**. +2. Install the snap-in in your DevRev by going to **Settings** > **Snap-ins** > **Install snap-in**. +3. Set up the connection under **Settings** > **Airdrops** > **Connections**. +4. Create an import at **Settings** > **Airdrops** > **Airdrop**. #### Observability From 0e73c47237d55184f7ae491b7e71be9cccad257f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Patricija=20Bre=C4=8Dko?= Date: Mon, 24 Mar 2025 13:30:39 +0100 Subject: [PATCH 04/27] Update overview --- fern/docs/pages/airdrop/overview.mdx | 102 ++++++++++++++++++++++++--- fern/versions/public.yml | 3 +- 2 files changed, 93 insertions(+), 12 deletions(-) diff --git a/fern/docs/pages/airdrop/overview.mdx b/fern/docs/pages/airdrop/overview.mdx index cdc4eaea..e4579c4e 100644 --- a/fern/docs/pages/airdrop/overview.mdx +++ b/fern/docs/pages/airdrop/overview.mdx @@ -1,6 +1,6 @@ _Airdrop_ is DevRev’s solution to migrate data. It allows our customers to bring in their existing data from external systems to DevRev, export data back to external systems, and keep data in sync -between DevRev and the external systems. +between DevRev and the external systems. It uses a standardized communication protocol for talking to Airdrop and a standardized data structure for all the files it extracts and processes, so that they can be seamlessly imported into DevRev. You can read more about Airdrop in the [general documentation](https://docs.devrev.ai/import#airdrop-features). DevRev is extending Airdrop's capabilities to give snap-in developers the ability to integrate with @@ -36,22 +36,102 @@ s3[(AWS S3)] ``` -An _extractor_ is a function in an Airdrop-capable snap-in responsible for extracting data from an -_external system_, such as Jira, Zendesk, or HubSpot. -It uses a standardized communication protocol for talking to Airdrop and a standardized data -structure for all the files it extracts and processes, so that they can be seamlessly imported -into DevRev. +## Terminology -## Sync runs +### Sync run -Airdrop functions are executed in the context of _sync runs_, which is a directed operation that -spans over many invocations of an Airdrop snap-in to bring the data to DevRev or load the data -to the external system. +Airdrop extractions are done in _sync runs_. +A sync run is one execution of an import or sync. + +### Forward sync A _forward sync_ is a sync run from an external system to DevRev. An extractor function in the snap-in is responsible for extracting data from the external system. +### Reverse sync + A _reverse sync_ is a sync run from DevRev to an external system. It uses a loader function, to create or update data in the external system. -TODO: 1-way sync, 2-way sync, incremental sync, ... +### 1-way (incremental) sync + +A _1-way sync_ (or _incremental sync_) refers to any extraction after the initial import has been successfully completed. +An extractor extracts data that was created or updated in the external system after the start +of the latest successful forward sync, including any changes that occurred during the forward sync, +but were not picked up by it. + +A snap-in must consult its state to get information on when the last successful forward sync started. +Airdrop snap-ins must maintain their own states that persists between phases in a sync run, +as well as between sync runs. + +A 1-way sync extracts only the domain objects updated or created since the previous successful sync run. + +## Sync run phases + +Each sync run is comprised out of phases. +Phases follow sequentially, and each can consist of one or more invocations of the snap-in. + +1. External sync units extraction +2. Metadata extraction +3. Data extraction +4. Attachments extraction + +```mermaid +sequenceDiagram +%%{init: { + 'theme': 'base', + 'themeVariables': { + 'fontFamily': 'Segoe UI', + 'signalColor': '#7d7f7c', + 'signalTextColor': '#7d7f7c' +}}}%% + actor user as User + participant gw as REST API Gateway + participant ad as Airdrop components + participant ee as External extractor + + note over gw,ad: Extract External Sync Units + user ->> gw: Select import connection + gw ->> ad: Start External Sync Unit extraction + ad ->> ee: Start External Sync Unit extraction + ee -->> gw: List of External Sync Units + gw ->> ad: List of External Sync Units + ad ->> gw: List of External Sync Units + gw -->> user: Show available External Sync Units + + + note over gw,ad: Extract Metadata + ad ->> ee: Start Metadata extraction + ee -->> gw: List of extracted metadata artifacts + gw ->> ad: List of extracted metadata artifacts + + note over gw,ad: Extract data + ad ->> ee: Start data extraction + ee -->> gw: List of extracted data artifacts + gw ->> ad: List of extracted data artifacts + + note over ad: Transform and import data into DevRev + + note over gw,ad: Extract attachments + ad ->> ee: Start attachment extraction + ee -->> gw: List of extracted attachments + gw ->> ad: List of extracted attachments + + ad -->> gw: Finished import + gw -->> user: Show finished import and report + + note over gw,ad: Delete data + user ->> gw: Delete import + gw ->> ad: Delete import + ad ->> ee: Delete data + ee -->> gw: Finished deleting data + gw ->> ad: Finished deleting data + + note over gw,ad: Delete attachments + ad ->> ee: Delete attachments + ee -->> gw: Finished deleting attachments + gw ->> ad: Finished deleting attachments + note over ad: Deletes internal data + ad -->> gw: Finished deleting import + gw -->> user: Import deleted +``` diff --git a/fern/versions/public.yml b/fern/versions/public.yml index 6c1572ab..d951b016 100644 --- a/fern/versions/public.yml +++ b/fern/versions/public.yml @@ -148,7 +148,7 @@ navigation: - page: "How does it work" slug: how-does-it-work path: ../docs/pages/airdrop/how-does-it-work.mdx - - page: "Snap-in examples" + - page: "Examples of snap-ins" slug: snap-in-examples path: ../docs/pages/airdrop/snap-in-examples.mdx - page: "Connecting to the external system" @@ -165,6 +165,7 @@ navigation: path: ../docs/pages/airdrop/loading-data.mdx - page: "Extraction phases" slug: extraction-phases + hidden: true path: ../docs/pages/airdrop/extraction-phases.mdx - page: "Loading phases" slug: loading-phases From 1b6813e48ab303280553d75e8bcbbc09b952d691 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Patricija=20Bre=C4=8Dko?= Date: Mon, 24 Mar 2025 13:39:55 +0100 Subject: [PATCH 05/27] Fix diagrams in overview --- fern/docs/pages/airdrop/overview.mdx | 1 - 1 file changed, 1 deletion(-) diff --git a/fern/docs/pages/airdrop/overview.mdx b/fern/docs/pages/airdrop/overview.mdx index e4579c4e..6e334632 100644 --- a/fern/docs/pages/airdrop/overview.mdx +++ b/fern/docs/pages/airdrop/overview.mdx @@ -33,7 +33,6 @@ s3[(AWS S3)] s3interact <-- Prepare upload/download URL --> s3 airdrop <-- Download/upload artifacts --> s3 end - ``` ## Terminology From 3fc1c3f0d5119b92494f2d123e4f013e155a143a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Patricija=20Bre=C4=8Dko?= Date: Mon, 24 Mar 2025 13:50:06 +0100 Subject: [PATCH 06/27] Add end to overview diagram --- fern/docs/pages/airdrop/overview.mdx | 1 + 1 file changed, 1 insertion(+) diff --git a/fern/docs/pages/airdrop/overview.mdx b/fern/docs/pages/airdrop/overview.mdx index 6e334632..4d6a5100 100644 --- a/fern/docs/pages/airdrop/overview.mdx +++ b/fern/docs/pages/airdrop/overview.mdx @@ -133,4 +133,5 @@ sequenceDiagram note over ad: Deletes internal data ad -->> gw: Finished deleting import gw -->> user: Import deleted + end ``` From 39d760e510701862b7191b67d08f3cafc49c06de Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Patricija=20Bre=C4=8Dko?= Date: Mon, 24 Mar 2025 14:00:17 +0100 Subject: [PATCH 07/27] Move some content from overview to how does it work --- fern/docs/pages/airdrop/how-does-it-work.mdx | 70 +++++++++++++++++++ fern/docs/pages/airdrop/overview.mdx | 71 -------------------- 2 files changed, 70 insertions(+), 71 deletions(-) diff --git a/fern/docs/pages/airdrop/how-does-it-work.mdx b/fern/docs/pages/airdrop/how-does-it-work.mdx index 109b179e..a656e22d 100644 --- a/fern/docs/pages/airdrop/how-does-it-work.mdx +++ b/fern/docs/pages/airdrop/how-does-it-work.mdx @@ -3,3 +3,73 @@ TODO: Explain how ADaaS code works by using the template TODO: Explain the role of manifest.yaml and where is what TODO: Explain how the communication between snap-in and DevRev works (through emitting events). + +## Sync run phases + +Each sync run is comprised out of phases. +Phases follow sequentially, and each can consist of one or more invocations of the snap-in. + +1. External sync units extraction +2. Metadata extraction +3. Data extraction +4. Attachments extraction + +```mermaid +sequenceDiagram +%%{init: { + 'theme': 'base', + 'themeVariables': { + 'fontFamily': 'Segoe UI', + 'signalColor': '#7d7f7c', + 'signalTextColor': '#7d7f7c' +}}}%% + actor user as User + participant gw as REST API Gateway + participant ad as Airdrop components + participant ee as External extractor + + note over gw,ad: Extract External Sync Units + user ->> gw: Select import connection + gw ->> ad: Start External Sync Unit extraction + ad ->> ee: Start External Sync Unit extraction + ee -->> gw: List of External Sync Units + gw ->> ad: List of External Sync Units + ad ->> gw: List of External Sync Units + gw -->> user: Show available External Sync Units + + + note over gw,ad: Extract Metadata + ad ->> ee: Start Metadata extraction + ee -->> gw: List of extracted metadata artifacts + gw ->> ad: List of extracted metadata artifacts + + note over gw,ad: Extract data + ad ->> ee: Start data extraction + ee -->> gw: List of extracted data artifacts + gw ->> ad: List of extracted data artifacts + + note over ad: Transform and import data into DevRev + + note over gw,ad: Extract attachments + ad ->> ee: Start attachment extraction + ee -->> gw: List of extracted attachments + gw ->> ad: List of extracted attachments + + ad -->> gw: Finished import + gw -->> user: Show finished import and report + + note over gw,ad: Delete data + user ->> gw: Delete import + gw ->> ad: Delete import + ad ->> ee: Delete data + ee -->> gw: Finished deleting data + gw ->> ad: Finished deleting data + + note over gw,ad: Delete attachments + ad ->> ee: Delete attachments + ee -->> gw: Finished deleting attachments + gw ->> ad: Finished deleting attachments + note over ad: Deletes internal data + ad -->> gw: Finished deleting import + gw -->> user: Import deleted +``` diff --git a/fern/docs/pages/airdrop/overview.mdx b/fern/docs/pages/airdrop/overview.mdx index 4d6a5100..dd18d57f 100644 --- a/fern/docs/pages/airdrop/overview.mdx +++ b/fern/docs/pages/airdrop/overview.mdx @@ -64,74 +64,3 @@ Airdrop snap-ins must maintain their own states that persists between phases in as well as between sync runs. A 1-way sync extracts only the domain objects updated or created since the previous successful sync run. - -## Sync run phases - -Each sync run is comprised out of phases. -Phases follow sequentially, and each can consist of one or more invocations of the snap-in. - -1. External sync units extraction -2. Metadata extraction -3. Data extraction -4. Attachments extraction - -```mermaid -sequenceDiagram -%%{init: { - 'theme': 'base', - 'themeVariables': { - 'fontFamily': 'Segoe UI', - 'signalColor': '#7d7f7c', - 'signalTextColor': '#7d7f7c' -}}}%% - actor user as User - participant gw as REST API Gateway - participant ad as Airdrop components - participant ee as External extractor - - note over gw,ad: Extract External Sync Units - user ->> gw: Select import connection - gw ->> ad: Start External Sync Unit extraction - ad ->> ee: Start External Sync Unit extraction - ee -->> gw: List of External Sync Units - gw ->> ad: List of External Sync Units - ad ->> gw: List of External Sync Units - gw -->> user: Show available External Sync Units - - - note over gw,ad: Extract Metadata - ad ->> ee: Start Metadata extraction - ee -->> gw: List of extracted metadata artifacts - gw ->> ad: List of extracted metadata artifacts - - note over gw,ad: Extract data - ad ->> ee: Start data extraction - ee -->> gw: List of extracted data artifacts - gw ->> ad: List of extracted data artifacts - - note over ad: Transform and import data into DevRev - - note over gw,ad: Extract attachments - ad ->> ee: Start attachment extraction - ee -->> gw: List of extracted attachments - gw ->> ad: List of extracted attachments - - ad -->> gw: Finished import - gw -->> user: Show finished import and report - - note over gw,ad: Delete data - user ->> gw: Delete import - gw ->> ad: Delete import - ad ->> ee: Delete data - ee -->> gw: Finished deleting data - gw ->> ad: Finished deleting data - - note over gw,ad: Delete attachments - ad ->> ee: Delete attachments - ee -->> gw: Finished deleting attachments - gw ->> ad: Finished deleting attachments - note over ad: Deletes internal data - ad -->> gw: Finished deleting import - gw -->> user: Import deleted - end -``` From 439746c87e15463ff649436ba4edb7037caf5943 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Patricija=20Bre=C4=8Dko?= Date: Mon, 24 Mar 2025 17:07:41 +0100 Subject: [PATCH 08/27] Update connecting to the external system page --- .../connecting-to-the-external-system.mdx | 66 ++++++++++++++++--- 1 file changed, 57 insertions(+), 9 deletions(-) diff --git a/fern/docs/pages/airdrop/connecting-to-the-external-system.mdx b/fern/docs/pages/airdrop/connecting-to-the-external-system.mdx index 21b98f71..702d3bd0 100644 --- a/fern/docs/pages/airdrop/connecting-to-the-external-system.mdx +++ b/fern/docs/pages/airdrop/connecting-to-the-external-system.mdx @@ -1,18 +1,66 @@ -TODO: Keyring explanation and guide setting up. - ## Keyrings -_Keyrings_ are a DevRev-specific mechanism for managing authentication to external systems. -They are called **Connections** in the DevRev app. +_Keyrings_ are a collection of authentication information, used by a snap-in to authenticate to the external system in API calls. This can include a key (e.g., a PAT token or API key), its type, the organization ID for which a key is valid, and in some cases the organization name. Keyrings provide a secure way to store and manage credentials within your DevRev snap-in. This eliminates the need to expose sensitive information like passwords or access tokens directly within your code or configuration files, enhancing overall security. They also provide a valid token by abstracting OAuth token renewal from the end user. -A keyring is used by a worker to authenticate to the external system in API calls. -They include the key (for example, a PAT or API key), its type, the organization ID for which a -key is valid, and in some cases the organization name. +They are called **Connections** in the DevRev app. + +### Configure a keyring + +Keyrings are configured in the `manifest.yaml` by configuring a `keyring_type`, like in the [example](https://github.com/devrev/adaas-template/blob/main/manifest.yaml). + +```yaml +keyring_types: + - id: + name: + description: + # The kind field specifies the type of keyring. + kind: <"secret"/"oauth2"> + # is_subdomain field specifies whether the keyring contains a subdomain. + # Enabling this field allows the keyring to get the subdomain from the user during creation. + # This is useful when the keyring requires a subdomain as part of the configuration. + # Default is false. + is_subdomain: + # Name of the external system you are importing from. + # This system name should start with a capital letter. + external_system_name: + secret_config: + # Define the fields in the secret. + # Each element represents one input field. + # User will provide this information when creating the connection in the UI. + # If omitted, the user will be asked for a generic secret. + fields: + - id: + name: + description: + is_optional: # optional: whether the field is optional or not. Default is false. + # The token_verification section is used to verify the token provided by the user. + token_verification: + # The URL to which the token will be sent for verification. + url: + # The HTTP method to be used for the verification request. + method: # The HTTP method to be used for the verification request. + # Optional: headers to be included in the verification request. + headers: + : + # Optional: query parameters to be included in the verification request. + query_params: + : # optional: query parameters to be included in the verification request. + # Fetching Organization Data: This allows you to retrieve additional information about the user's organization. + organization_data: + type: "config" + # The URL to which the request is sent to fetch organization data. + url: + # The HTTP method used to send the request. + method: "GET" + headers: + : + # The jq filter used to extract the organization data from the response. + response_jq: +``` -Configure a keyring for the external system in the `manifest.yaml`. -Refer to [Keyrings](/snapin-development/references/manifest#developer-keyrings) for instructions. +You can find more information about keyrings and keyring types [here](/snapin-development/references/keyrings/keyring-intro). From 7c49abd499d62aa0bececca30757908a99195597 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Patricija=20Bre=C4=8Dko?= Date: Tue, 25 Mar 2025 09:16:44 +0100 Subject: [PATCH 09/27] Update Getting started --- fern/docs/pages/airdrop/getting-started.mdx | 64 +++++++++++++++------ fern/versions/public.yml | 6 +- 2 files changed, 51 insertions(+), 19 deletions(-) diff --git a/fern/docs/pages/airdrop/getting-started.mdx b/fern/docs/pages/airdrop/getting-started.mdx index 95aaff6c..b9adcc44 100644 --- a/fern/docs/pages/airdrop/getting-started.mdx +++ b/fern/docs/pages/airdrop/getting-started.mdx @@ -2,7 +2,7 @@ ### Create a new organization -If this is your first time setting up an organization within DevRev, you'll need to create a dedicated DevRev organization for developing your new Airdrop snap-in: +If this is your first time setting up an organization within DevRev, you need to create a dedicated DevRev organization for developing your new Airdrop snap-in: 1. Click on your profile picture in the top left corner. 2. Go to the **Orgs** section and click on **+**. @@ -10,7 +10,11 @@ If this is your first time setting up an organization within DevRev, you'll need You now find your newly created organization under **Orgs**. -### Install required tools and packages +## Local development + +For ease of development, you can run your snap-in locally and receive logs directly in your console. + +### Prerequisites - [devrev-cli](https://developer.devrev.ai/snapin-development/references/cli-install) (version 4.7 or higher) - [jq](https://stedolan.github.io/jq) @@ -18,27 +22,55 @@ You now find your newly created organization under **Orgs**. - [nodejs](https://nodejs.org/en/download/package-manager) (version 18.x.x+ or higher) - [ngrok](https://ngrok.com) -## Setting up the snap-in - ### Starter template -DevRev provides a starter template for Airdrop snap-in development. - -1. Create a new repository from [Airdrop template repository](https://github.com/devrev/adaas-template) by selecting _Use this template_ and then _Create a new repository_. -2. Copy `.env.example` to a new file named `.env` and fill in the required variables. - -### Local development - -For ease of development, you can run your snap-in locally and receive logs directly in your console. Follow the guide [here](/snapin-development/locally-testing-snap-ins). +DevRev provides a template with example code to implement an Airdrop as a Service (ADaaS) snap-in. + +1. Create a new repository: + - Create a new repository from this template by clicking the "Use this template" button in the upper right corner and then "Create a new repository". + - The repository name must start with `airdrop-` (e.g., `airdrop--snap-in`). +2. Open the project in your IDE and authenticate to your DevRev organization using the DevRev CLI: + ```bash + devrev profiles authenticate --org + ``` +3. Install the snap-in dependencies from the `code` directory: + ```bash + npm ci + ``` +4. Start the snap-in development server from the `code` directory: + ```bash + npm run test:server -- local + ``` + This will start the snap-in server on `http://localhost:8000`. +5. Start the ngrok tunnel in a separate terminal window: + ```bash + ngrok http 8000 + ``` + This will create a tunnel to your local server. + The ngrok forwarding URL will be displayed in the terminal window. +6. Copy the ngrok forwarding URL and create a new snap-in version and snap-in package using the DevRev CLI: + ```bash + devrev snap_in_version create-one --manifest ./manifest.yaml --create-package --testing-url + ``` +7. Create a snap-in draft using the DevRev CLI: + ```bash + devrev snap_in draft + ``` +8. After the snap-in draft is created, install the snap-in: + ```bash + devrev snap_in activate + ``` +9. Start the import (**Airdrops** -> **Start Airdrop** -> **your snap-in**). ### Publishing the snap-in Once you're ready to launch your snap-in in a production environment, follow these steps: -1. Deploy a draft version of your snap-in to your organization by using `make deploy`. -2. Install the snap-in in your DevRev by going to **Settings** > **Snap-ins** > **Install snap-in**. -3. Set up the connection under **Settings** > **Airdrops** > **Connections**. -4. Create an import at **Settings** > **Airdrops** > **Airdrop**. +1. Copy `.env.example` to a new file named `.env` and fill in the required variables. +2. Deploy a draft version of your snap-in to your organization by using `make deploy`. +3. Install the snap-in in your DevRev by going to **Settings** > **Snap-ins** > **Install snap-in**. +4. Set up the connection under **Settings** > **Airdrops** > **Connections**. +5. Create an import at **Settings** > **Airdrops** > **Airdrop**. #### Observability diff --git a/fern/versions/public.yml b/fern/versions/public.yml index d951b016..10a77c78 100644 --- a/fern/versions/public.yml +++ b/fern/versions/public.yml @@ -151,12 +151,12 @@ navigation: - page: "Examples of snap-ins" slug: snap-in-examples path: ../docs/pages/airdrop/snap-in-examples.mdx - - page: "Connecting to the external system" - slug: connecting-to-the-external-system - path: ../docs/pages/airdrop/connecting-to-the-external-system.mdx - section: "Development guide" slug: development-guide contents: + - page: "Connecting to the external system" + slug: connecting-to-the-external-system + path: ../docs/pages/airdrop/connecting-to-the-external-system.mdx - page: "Extracting data from external system" slug: extracting-data path: ../docs/pages/airdrop/extracting-data.mdx From 1c23f3e174a74f8f627688e2171141a6ceb8a63b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Patricija=20Bre=C4=8Dko?= Date: Tue, 25 Mar 2025 09:57:51 +0100 Subject: [PATCH 10/27] Updates --- .../connecting-to-the-external-system.mdx | 2 +- fern/docs/pages/airdrop/getting-started.mdx | 6 +- fern/docs/pages/airdrop/how-does-it-work.mdx | 75 ------------------- .../airdrop/how-does-the-snap-in-work.mdx | 65 ++++++++++++++++ fern/docs/pages/airdrop/overview.mdx | 32 +------- fern/versions/public.yml | 18 ++--- 6 files changed, 81 insertions(+), 117 deletions(-) delete mode 100644 fern/docs/pages/airdrop/how-does-it-work.mdx create mode 100644 fern/docs/pages/airdrop/how-does-the-snap-in-work.mdx diff --git a/fern/docs/pages/airdrop/connecting-to-the-external-system.mdx b/fern/docs/pages/airdrop/connecting-to-the-external-system.mdx index 702d3bd0..e2d776b6 100644 --- a/fern/docs/pages/airdrop/connecting-to-the-external-system.mdx +++ b/fern/docs/pages/airdrop/connecting-to-the-external-system.mdx @@ -1,6 +1,6 @@ ## Keyrings -_Keyrings_ are a collection of authentication information, used by a snap-in to authenticate to the external system in API calls. This can include a key (e.g., a PAT token or API key), its type, the organization ID for which a key is valid, and in some cases the organization name. +_Keyrings_ are a collection of authentication information, used by a snap-in to authenticate to the external system in API calls. This can include a key (for example, a PAT token or API key), its type, the organization ID for which a key is valid, and in some cases the organization name. Keyrings provide a secure way to store and manage credentials within your DevRev snap-in. This eliminates the need to expose sensitive information like passwords or access tokens directly diff --git a/fern/docs/pages/airdrop/getting-started.mdx b/fern/docs/pages/airdrop/getting-started.mdx index b9adcc44..7dc03a20 100644 --- a/fern/docs/pages/airdrop/getting-started.mdx +++ b/fern/docs/pages/airdrop/getting-started.mdx @@ -60,9 +60,9 @@ DevRev provides a template with example code to implement an Airdrop as a Servic ```bash devrev snap_in activate ``` -9. Start the import (**Airdrops** -> **Start Airdrop** -> **your snap-in**). +9. Start the import (**Airdrops** -> **Start Airdrop** -> **Your snap-in**). -### Publishing the snap-in +## Publishing the snap-in Once you're ready to launch your snap-in in a production environment, follow these steps: @@ -72,7 +72,7 @@ Once you're ready to launch your snap-in in a production environment, follow the 4. Set up the connection under **Settings** > **Airdrops** > **Connections**. 5. Create an import at **Settings** > **Airdrops** > **Airdrop**. -#### Observability +### Observability To observe logs from your snap-in in your development environment: diff --git a/fern/docs/pages/airdrop/how-does-it-work.mdx b/fern/docs/pages/airdrop/how-does-it-work.mdx deleted file mode 100644 index a656e22d..00000000 --- a/fern/docs/pages/airdrop/how-does-it-work.mdx +++ /dev/null @@ -1,75 +0,0 @@ -TODO: Explain how ADaaS code works by using the template - -TODO: Explain the role of manifest.yaml and where is what - -TODO: Explain how the communication between snap-in and DevRev works (through emitting events). - -## Sync run phases - -Each sync run is comprised out of phases. -Phases follow sequentially, and each can consist of one or more invocations of the snap-in. - -1. External sync units extraction -2. Metadata extraction -3. Data extraction -4. Attachments extraction - -```mermaid -sequenceDiagram -%%{init: { - 'theme': 'base', - 'themeVariables': { - 'fontFamily': 'Segoe UI', - 'signalColor': '#7d7f7c', - 'signalTextColor': '#7d7f7c' -}}}%% - actor user as User - participant gw as REST API Gateway - participant ad as Airdrop components - participant ee as External extractor - - note over gw,ad: Extract External Sync Units - user ->> gw: Select import connection - gw ->> ad: Start External Sync Unit extraction - ad ->> ee: Start External Sync Unit extraction - ee -->> gw: List of External Sync Units - gw ->> ad: List of External Sync Units - ad ->> gw: List of External Sync Units - gw -->> user: Show available External Sync Units - - - note over gw,ad: Extract Metadata - ad ->> ee: Start Metadata extraction - ee -->> gw: List of extracted metadata artifacts - gw ->> ad: List of extracted metadata artifacts - - note over gw,ad: Extract data - ad ->> ee: Start data extraction - ee -->> gw: List of extracted data artifacts - gw ->> ad: List of extracted data artifacts - - note over ad: Transform and import data into DevRev - - note over gw,ad: Extract attachments - ad ->> ee: Start attachment extraction - ee -->> gw: List of extracted attachments - gw ->> ad: List of extracted attachments - - ad -->> gw: Finished import - gw -->> user: Show finished import and report - - note over gw,ad: Delete data - user ->> gw: Delete import - gw ->> ad: Delete import - ad ->> ee: Delete data - ee -->> gw: Finished deleting data - gw ->> ad: Finished deleting data - - note over gw,ad: Delete attachments - ad ->> ee: Delete attachments - ee -->> gw: Finished deleting attachments - gw ->> ad: Finished deleting attachments - note over ad: Deletes internal data - ad -->> gw: Finished deleting import - gw -->> user: Import deleted -``` diff --git a/fern/docs/pages/airdrop/how-does-the-snap-in-work.mdx b/fern/docs/pages/airdrop/how-does-the-snap-in-work.mdx new file mode 100644 index 00000000..81532b3c --- /dev/null +++ b/fern/docs/pages/airdrop/how-does-the-snap-in-work.mdx @@ -0,0 +1,65 @@ +## How does it work? + +A snap-in is a software that runs in Node.js and communicates with Airdrop. + +It uses a standardized communication protocol for talking to Airdrop and a standardized data structure for all the files it extracts and processes, so that they can be seamlessly imported into DevRev. + +A sync run is comprised of 4 phases, which follow sequentially: + +1. External sync units extraction (only on initial import) +2. Metadata extraction +3. Data extraction +4. Attachments extraction + +### External sync units extraction + +In the external sync unit extraction phase, the snap-in is expected to obtain a list of external +sync units that it can extract from the external system API and send it to Airdrop in its response. + +An _external sync unit_ refers to a single unit in the external system that is being airdropped to DevRev. +In some systems, this is a project; in some it is a repository; in support systems it could be +called a brand or an organization. +What a unit of data is called and what it represents depends on the external system's domain model. +It usually combines contacts, users, work-like items, and comments into a unit of domain objects. + +Some external systems may offer a single unit in their free plans, +while their enterprise plans may offer their clients to operate many separate units. + +The external sync unit ID is the identifier of the sync unit (project, repository, or similar) +in the external system. +For GitHub, this would be the repository, for example `cli` in `github.com/devrev/cli`. + +### Metadata extraction + +During the metadata extraction phase, the Airdrop snap-in must provide an +`external_domain_metadata.json` file on each sync run. +This file provides a structured way of describing the external system's domain system, +including its domain entities, types, relationships, and other metadata. + +### Data extraction + +In the data extraction phase, the extractor is expected to call the external system's APIs +to retrieve all the items that were updated since the start of the last extraction. +If there was no previous extraction (the current run is an initial import), +then all the items should be extracted. + +### Attachments extraction + +For the attachment extraction phase of the import process, the extractor has to upload each +attachment to DevRev + +## Recommended steps of snap-in development + +1. Inspect available objects in the external system and in DevRev - think about what will map to what +2. Check which fields in the objects are required by DevRev (Where should I check that? - API docs? Schema?) and which fields are available in the external system API. +3. Decide on what will be the external sync unit (How do I decide? An external sync unit should be the smallest unit containing the objects you want to import (e.g. a project).) +4. Implement External sync units extraction (check below) + Prepare API endpoints from the external system and get data that we will be importing in .json file for each object type (tasks.json, issues.json, users.json). + https://github.com/devrev/adaas-chef-cli/blob/main/docs/step_by_step.md#getting-a-good-starting-point-metadata-using-the-infer-metadata-command +5. Implement Metadata extraction (check below) + +TODO: Explain how ADaaS code works by using the template + +TODO: Explain the role of manifest.yaml and where is what + +TODO: Explain how the communication between snap-in and DevRev works (through emitting events). diff --git a/fern/docs/pages/airdrop/overview.mdx b/fern/docs/pages/airdrop/overview.mdx index dd18d57f..1b56915b 100644 --- a/fern/docs/pages/airdrop/overview.mdx +++ b/fern/docs/pages/airdrop/overview.mdx @@ -1,40 +1,14 @@ _Airdrop_ is DevRev’s solution to migrate data. It allows our customers to bring in their existing data from external systems to DevRev, export data back to external systems, and keep data in sync -between DevRev and the external systems. It uses a standardized communication protocol for talking to Airdrop and a standardized data structure for all the files it extracts and processes, so that they can be seamlessly imported into DevRev. -You can read more about Airdrop in the [general documentation](https://docs.devrev.ai/import#airdrop-features). +between DevRev and the external systems. + +You can learn more about Airdrop in the [general documentation](https://docs.devrev.ai/import#airdrop-features). DevRev is extending Airdrop's capabilities to give snap-in developers the ability to integrate with DevRev’s Airdrop functionality. It enables developers to create external workers (extractors and loaders) to bring data from various external systems. -```mermaid -flowchart TD -%%{init: { - 'theme': 'base', - 'themeVariables': { - 'fontFamily': 'Segoe UI', - 'lineColor': '#7d7f7c' -}}}%% -externalSystem[External system] -worker([Worker]) -s3interact[S3interact] -airdrop[Airdrop] -s3[(AWS S3)] - - externalSystem <-- Get/create users, issues, ... --> worker - - worker <-- Exchange artifact upload/download URLs --> s3interact - worker <-- Upload/download artifacts --> s3 - worker <-- Airdrop messages and REST API --> airdrop - - - subgraph DevRev - s3interact <-- Prepare upload/download URL --> s3 - airdrop <-- Download/upload artifacts --> s3 - end -``` - ## Terminology ### Sync run diff --git a/fern/versions/public.yml b/fern/versions/public.yml index 10a77c78..1b350327 100644 --- a/fern/versions/public.yml +++ b/fern/versions/public.yml @@ -142,18 +142,15 @@ navigation: - page: "Overview" slug: overview path: ../docs/pages/airdrop/overview.mdx - - page: "Getting started" - slug: getting-started - path: ../docs/pages/airdrop/getting-started.mdx - - page: "How does it work" - slug: how-does-it-work - path: ../docs/pages/airdrop/how-does-it-work.mdx - - page: "Examples of snap-ins" - slug: snap-in-examples - path: ../docs/pages/airdrop/snap-in-examples.mdx - section: "Development guide" slug: development-guide contents: + - page: "How does a snap-in work" + slug: how-does-the-snap-in-work + path: ../docs/pages/airdrop/how-does-the-snap-in-work.mdx + - page: "Getting started" + slug: getting-started + path: ../docs/pages/airdrop/getting-started.mdx - page: "Connecting to the external system" slug: connecting-to-the-external-system path: ../docs/pages/airdrop/connecting-to-the-external-system.mdx @@ -163,6 +160,9 @@ navigation: - page: "Loading data to DevRev" slug: loading-data path: ../docs/pages/airdrop/loading-data.mdx + - page: "Examples of snap-ins" + slug: snap-in-examples + path: ../docs/pages/airdrop/snap-in-examples.mdx - page: "Extraction phases" slug: extraction-phases hidden: true From 626be242d0de9fa3f2ce975e3e47bcc1c7f67a87 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Patricija=20Bre=C4=8Dko?= Date: Tue, 25 Mar 2025 15:16:26 +0100 Subject: [PATCH 11/27] Update how does it work section --- .../airdrop/how-does-the-snap-in-work.mdx | 91 ++++++++++++++----- fern/docs/pages/airdrop/overview.mdx | 4 +- fern/versions/public.yml | 6 +- 3 files changed, 74 insertions(+), 27 deletions(-) diff --git a/fern/docs/pages/airdrop/how-does-the-snap-in-work.mdx b/fern/docs/pages/airdrop/how-does-the-snap-in-work.mdx index 81532b3c..ede1efde 100644 --- a/fern/docs/pages/airdrop/how-does-the-snap-in-work.mdx +++ b/fern/docs/pages/airdrop/how-does-the-snap-in-work.mdx @@ -1,17 +1,76 @@ -## How does it work? +## Snap-in code structure -A snap-in is a software that runs in Node.js and communicates with Airdrop. +A snap-in is a Node.js project with a specific structure: -It uses a standardized communication protocol for talking to Airdrop and a standardized data structure for all the files it extracts and processes, so that they can be seamlessly imported into DevRev. +``` +- manifest.yaml +- code/ + - scripts/ + - src/ + - fixtures/ + - functions/ + - external-system/ + - extraction/ + - workers/ + - attachments-extraction.ts + - data-extraction.ts + - external-sync-units-extraction.ts + - metadata-extraction.ts + - index.ts + - install_initial_domain_mapping/ + - loading/ + - workers/ + - load-data.ts + - data-extraction.ts + - external-sync-units-extraction.ts + - metadata-extraction.ts + - index.ts + - test-runner/ + - test/ +``` -A sync run is comprised of 4 phases, which follow sequentially: +#### `manifest.yaml` + +The snap-in manifest an essential part of a snap-in, as it's used to define a snap-in configuration: snap-in name, description, version, keyring types and functionality (functions). + +#### `code/` + +This folder contains the Node.js project. + +#### `scripts/` + +This folder includes scripts for deploying (`deploy.sh`) and building (`build.sh`) the snap-in using a Makefile. + +#### `functions/` + +This directory holds the snap-in's core logic. The logic for extracting data from the external system is found in the `extraction` folder, and the logic for loading data into the external system from DevRev is in the `loading` folder. + +## Snap-in functionality + +The DevRev platform enables the creation of a snap-in that can either import data from an external system into DevRev or export data from DevRev to an external system. Implementing both isn't necessary. The implemented functionalities are specified in `manifest.yaml` under the `functions` key, as shown in the starter template: + +```yaml +functions: + - name: extraction + description: Extraction function for the template snap-in + - name: loading + description: Loading function for the template snap-in + - name: install_initial_domain_mapping + description: Create blueprint and install initial domain mapping +``` + +For example, removing the entry with the name loading means syncing from DevRev to the external system is not available in the UI. + +### Extraction + +Extraction is comprised of 4 phases, which follow sequentially: 1. External sync units extraction (only on initial import) 2. Metadata extraction 3. Data extraction 4. Attachments extraction -### External sync units extraction +#### External sync units extraction In the external sync unit extraction phase, the snap-in is expected to obtain a list of external sync units that it can extract from the external system API and send it to Airdrop in its response. @@ -29,37 +88,25 @@ The external sync unit ID is the identifier of the sync unit (project, repositor in the external system. For GitHub, this would be the repository, for example `cli` in `github.com/devrev/cli`. -### Metadata extraction +#### Metadata extraction During the metadata extraction phase, the Airdrop snap-in must provide an `external_domain_metadata.json` file on each sync run. This file provides a structured way of describing the external system's domain system, including its domain entities, types, relationships, and other metadata. -### Data extraction +#### Data extraction In the data extraction phase, the extractor is expected to call the external system's APIs to retrieve all the items that were updated since the start of the last extraction. If there was no previous extraction (the current run is an initial import), then all the items should be extracted. -### Attachments extraction +#### Attachments extraction For the attachment extraction phase of the import process, the extractor has to upload each attachment to DevRev -## Recommended steps of snap-in development - -1. Inspect available objects in the external system and in DevRev - think about what will map to what -2. Check which fields in the objects are required by DevRev (Where should I check that? - API docs? Schema?) and which fields are available in the external system API. -3. Decide on what will be the external sync unit (How do I decide? An external sync unit should be the smallest unit containing the objects you want to import (e.g. a project).) -4. Implement External sync units extraction (check below) - Prepare API endpoints from the external system and get data that we will be importing in .json file for each object type (tasks.json, issues.json, users.json). - https://github.com/devrev/adaas-chef-cli/blob/main/docs/step_by_step.md#getting-a-good-starting-point-metadata-using-the-infer-metadata-command -5. Implement Metadata extraction (check below) - -TODO: Explain how ADaaS code works by using the template - -TODO: Explain the role of manifest.yaml and where is what +### Loading -TODO: Explain how the communication between snap-in and DevRev works (through emitting events). +TODO diff --git a/fern/docs/pages/airdrop/overview.mdx b/fern/docs/pages/airdrop/overview.mdx index 1b56915b..b721c57c 100644 --- a/fern/docs/pages/airdrop/overview.mdx +++ b/fern/docs/pages/airdrop/overview.mdx @@ -2,8 +2,6 @@ _Airdrop_ is DevRev’s solution to migrate data. It allows our customers to bri data from external systems to DevRev, export data back to external systems, and keep data in sync between DevRev and the external systems. -You can learn more about Airdrop in the [general documentation](https://docs.devrev.ai/import#airdrop-features). - DevRev is extending Airdrop's capabilities to give snap-in developers the ability to integrate with DevRev’s Airdrop functionality. It enables developers to create external workers (extractors and loaders) to bring data from various @@ -38,3 +36,5 @@ Airdrop snap-ins must maintain their own states that persists between phases in as well as between sync runs. A 1-way sync extracts only the domain objects updated or created since the previous successful sync run. + +You can learn more about Airdrop in the [general documentation](https://docs.devrev.ai/import#airdrop-features). diff --git a/fern/versions/public.yml b/fern/versions/public.yml index 1b350327..06f8ac43 100644 --- a/fern/versions/public.yml +++ b/fern/versions/public.yml @@ -145,12 +145,12 @@ navigation: - section: "Development guide" slug: development-guide contents: - - page: "How does a snap-in work" - slug: how-does-the-snap-in-work - path: ../docs/pages/airdrop/how-does-the-snap-in-work.mdx - page: "Getting started" slug: getting-started path: ../docs/pages/airdrop/getting-started.mdx + - page: "How does a snap-in work" + slug: how-does-the-snap-in-work + path: ../docs/pages/airdrop/how-does-the-snap-in-work.mdx - page: "Connecting to the external system" slug: connecting-to-the-external-system path: ../docs/pages/airdrop/connecting-to-the-external-system.mdx From 06cb02f84c77a413615ab1855422235ae70e9f72 Mon Sep 17 00:00:00 2001 From: GasperSenk Date: Tue, 25 Mar 2025 15:54:02 +0100 Subject: [PATCH 12/27] musings on state --- fern/docs/pages/airdrop/state-handling.mdx | 54 ++++++++++++++++++++++ 1 file changed, 54 insertions(+) create mode 100644 fern/docs/pages/airdrop/state-handling.mdx diff --git a/fern/docs/pages/airdrop/state-handling.mdx b/fern/docs/pages/airdrop/state-handling.mdx new file mode 100644 index 00000000..894d4180 --- /dev/null +++ b/fern/docs/pages/airdrop/state-handling.mdx @@ -0,0 +1,54 @@ +# Snap-in `state` + +Since each snap-in invocation is a separate runtime instance, it does not know what has been previously accomplished. + +To enable information passing between invocations and runs we have added support for saving a limited amount of data as snap-in `state`. + +* Snap-in's `state` is loaded at the start of each invocation and saved at its end. +* Each sync direction (to DevRev and from DevRev) has its own `state` object that is not shared. +* Snap-in `state` should be smaller than 1 MB.Hard cap is a bit higher. 1 MB maps to approximately 500 000 characters. + +Effective use of the `state` and breaking down the problem into smaller chunks is crucial for good performance and user experience. + +Without knowing what has been processed, the snap-in will extract the same data multiple times, using precious API capacity, time, and posibly duplicate the data inside DevRev or the external application. Duplicate data will lead to frustrating snap-in user experience. + +# Data queries + +First step is to research the external system of records (SOR) public API capabilities. + +* Does it offer searching on created date, modified date, some append only change log? +* How do records relate to each other, can I query them independantly? +* Do records contain parent and/or children information. +* Does a list call return only a summary or all needed information. Do I need to create separate calls to retrieve more than X comments, or threaded comments? +* How many documents does a list call return? Is it best to limit them due to low API limits? + +# Example system + +Lets define the limits of a theoretical system: + +* Allows list queries by the changed date that returns 50 items in one batch. +* Each item contains up to 10 comments, for more you need to use a different API endpoint. +* Retrieved item contains a flag indicating the existence of attachments. + +This simple system has a straightforward data template with a few API limits. + +* Parent item can be retrieved with a list call. +* If the parent contains 10 comments, we have to perform a separate call to see if more comments exist. +* In the separate attachment phase we have to list though items again and extract attachments from those that exist. +* We cannot save all the IDs of parent objects with attachments as that could breach `state` size limits. +* It is best for the `state` to contain last changed date timestamp plus some item IDs in case multiple items were changed at the same time. All 50 item IDs could easily fit in the `state`. +* Between list calls some items can be updated. Two queries with the same changed date might not return the same items. + +# Set the `state` + +In our theoretical system we would be able to batch calls in the increments of 50. + +* In each batch we would extract items and all of their comments. + +After each batch is processed, changed date, item IDs and the changed date of the last seen item is saved in the `state`. +In the next batch we can use the changed date of the last seen item as the starting point. If we are afraid to lose some items, we can always decrement the smallest time chunk the platform supports to make sure some items overlap from one batch to another. +We also have to make sure to not trap ourselves into the cycle to always call with the same changed date and forever retrieve the same items. + +In the same way as we determined to use changed date as a discriminator between chunks we have to make sure we can use an indicator to continue between invocation and between sync runs. + +**As we have to list through all items to extract attachments, we need to make sure that we do not override the changed date and item IDs state between attachment and data phase themselves**. \ No newline at end of file From 9aa52c79763f9a5c7c8c6af6e734883bb3707029 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Patricija=20Bre=C4=8Dko?= Date: Tue, 25 Mar 2025 15:56:27 +0100 Subject: [PATCH 13/27] Update publish to marketplace --- fern/docs/pages/airdrop/getting-started.mdx | 2 +- .../airdrop/how-does-the-snap-in-work.mdx | 20 ++++----- fern/docs/pages/airdrop/overview.mdx | 2 + .../pages/airdrop/publish-to-marketplace.mdx | 42 ++++++++++++++++++- 4 files changed, 53 insertions(+), 13 deletions(-) diff --git a/fern/docs/pages/airdrop/getting-started.mdx b/fern/docs/pages/airdrop/getting-started.mdx index 7dc03a20..848136c2 100644 --- a/fern/docs/pages/airdrop/getting-started.mdx +++ b/fern/docs/pages/airdrop/getting-started.mdx @@ -28,7 +28,7 @@ DevRev provides a template with example code to implement an Airdrop as a Servic 1. Create a new repository: - Create a new repository from this template by clicking the "Use this template" button in the upper right corner and then "Create a new repository". - - The repository name must start with `airdrop-` (e.g., `airdrop--snap-in`). + - The repository name must start with `airdrop-` (for example, `airdrop--snap-in`). 2. Open the project in your IDE and authenticate to your DevRev organization using the DevRev CLI: ```bash devrev profiles authenticate --org diff --git a/fern/docs/pages/airdrop/how-does-the-snap-in-work.mdx b/fern/docs/pages/airdrop/how-does-the-snap-in-work.mdx index ede1efde..27ee0cb6 100644 --- a/fern/docs/pages/airdrop/how-does-the-snap-in-work.mdx +++ b/fern/docs/pages/airdrop/how-does-the-snap-in-work.mdx @@ -29,6 +29,8 @@ A snap-in is a Node.js project with a specific structure: - test/ ``` +TODO: describe emitting events and invoking workers? + #### `manifest.yaml` The snap-in manifest an essential part of a snap-in, as it's used to define a snap-in configuration: snap-in name, description, version, keyring types and functionality (functions). @@ -65,10 +67,10 @@ For example, removing the entry with the name loading means syncing from DevRev Extraction is comprised of 4 phases, which follow sequentially: -1. External sync units extraction (only on initial import) -2. Metadata extraction -3. Data extraction -4. Attachments extraction +1. external sync units extraction (only on initial import) +2. metadata extraction +3. data extraction +4. attachments extraction #### External sync units extraction @@ -97,15 +99,13 @@ including its domain entities, types, relationships, and other metadata. #### Data extraction -In the data extraction phase, the extractor is expected to call the external system's APIs -to retrieve all the items that were updated since the start of the last extraction. -If there was no previous extraction (the current run is an initial import), -then all the items should be extracted. +In initial sync we need to extract all data to create a baseline while in subsequent runs only updated objects need to be extracted. + +Snap-in should consult its internal state to determine what is already extracted and what it did not #### Attachments extraction -For the attachment extraction phase of the import process, the extractor has to upload each -attachment to DevRev +For the attachment extraction phase of the import process, the extractor has to upload each attachment to DevRev ### Loading diff --git a/fern/docs/pages/airdrop/overview.mdx b/fern/docs/pages/airdrop/overview.mdx index b721c57c..226d6353 100644 --- a/fern/docs/pages/airdrop/overview.mdx +++ b/fern/docs/pages/airdrop/overview.mdx @@ -37,4 +37,6 @@ as well as between sync runs. A 1-way sync extracts only the domain objects updated or created since the previous successful sync run. +TODO: define the sync unit and how it relates to external project/sync unit, ... + You can learn more about Airdrop in the [general documentation](https://docs.devrev.ai/import#airdrop-features). diff --git a/fern/docs/pages/airdrop/publish-to-marketplace.mdx b/fern/docs/pages/airdrop/publish-to-marketplace.mdx index 76f90301..ca0f731c 100644 --- a/fern/docs/pages/airdrop/publish-to-marketplace.mdx +++ b/fern/docs/pages/airdrop/publish-to-marketplace.mdx @@ -1,3 +1,41 @@ -When you feel like your snap-in is ready to be used by different organizations, you can publish it to the DevRev marketplace. +When your snap-in is ready for use by you or other organizations, publish it to the DevRev marketplace. Before starting the publication process, ensure your snap-in is deployed to your organization. If you need a refresher, refer to the [instructions](/snapin-development/adaas/development-guide/getting-started#publishing-the-snap-in) for deploying your snap-in. -Follow this [guide](/snapin-development/marketplace-listings). +### 1. Create marketplace listing + +When creating a new marketplace listing, an initial marketplace submission must be created with all the mandatory properties. To create a new listing, run the following command, which will start a wizard to guide you through the process: + +```bash +devrev marketplace_submissions create +``` + +### 2. Submit marketplace listing for review + +The newly created marketplace submission is in the Draft state. In order to publish it, we first have to transition it to Waiting for review state. This can be done by running the following command, which will start a wizard to guide you through the process: + +```bash +devrev marketplace_submissions transition +``` + +Once the submission is transitioned to the Waiting for review state, it needs to be approved by a marketplace admin. While in review, the state of the submission is In review. Once it is reviewed, the state will be changed to either Approved or Rejected. + + + Submissions whose state is _In review_ or _Approved_ cannot be modified. + + +If the submission was rejected, you can transition it back to Draft and modify it to satisfy the requirements. Once it is updated, you can transition it back to Waiting for review to be reviewed again. + +### 3. Publish marketplace submission + +If the submission was approved, you can publish it using the following command, which will start a wizard to guide you through the process: + +```bash +devrev marketplace_items publish +``` + +To make sure that marketplace item was published you can retrieve it using its ID: + +```bash +devrev marketplace_items show [marketplace_item_id] | jq '{name: .name, id: .id, state: .state}' +``` + +Learn more about the markeplace [here](/snapin-development/marketplace-listings). From b6e64a88b4cfbb61bc6e80a823c5bddb02c13b00 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Patricija=20Bre=C4=8Dko?= Date: Wed, 26 Mar 2025 11:23:54 +0100 Subject: [PATCH 14/27] Add first steps of development --- fern/docs/pages/airdrop/development-guide.mdx | 7 -- fern/docs/pages/airdrop/extracting-data.mdx | 46 +++++++- .../airdrop/first-steps-of-development.mdx | 64 +++++++++++ .../airdrop/how-does-the-snap-in-work.mdx | 104 +++++------------- fern/docs/pages/airdrop/overview.mdx | 6 +- fern/versions/public.yml | 51 ++++----- 6 files changed, 161 insertions(+), 117 deletions(-) delete mode 100644 fern/docs/pages/airdrop/development-guide.mdx create mode 100644 fern/docs/pages/airdrop/first-steps-of-development.mdx diff --git a/fern/docs/pages/airdrop/development-guide.mdx b/fern/docs/pages/airdrop/development-guide.mdx deleted file mode 100644 index 7104b493..00000000 --- a/fern/docs/pages/airdrop/development-guide.mdx +++ /dev/null @@ -1,7 +0,0 @@ -## Extracting data from external system - -TODO: Explain the order of development steps. - -## Loading data to DevRev - -TODO: Explain the order of development steps. diff --git a/fern/docs/pages/airdrop/extracting-data.mdx b/fern/docs/pages/airdrop/extracting-data.mdx index 1333ed77..0a4763c0 100644 --- a/fern/docs/pages/airdrop/extracting-data.mdx +++ b/fern/docs/pages/airdrop/extracting-data.mdx @@ -1 +1,45 @@ -TODO +## Snap-in functionality + +### Extraction + +Extraction is comprised of 4 phases, which follow sequentially: + +1. external sync units extraction (only on initial import) +2. metadata extraction +3. data extraction +4. attachments extraction + +#### External sync units extraction + +In the external sync unit extraction phase, the snap-in is expected to obtain a list of external +sync units that it can extract from the external system API and send it to Airdrop in its response. + +An _external sync unit_ refers to a single unit in the external system that is being airdropped to DevRev. +In some systems, this is a project; in some it is a repository; in support systems it could be +called a brand or an organization. +What a unit of data is called and what it represents depends on the external system's domain model. +It usually combines contacts, users, work-like items, and comments into a unit of domain objects. + +Some external systems may offer a single unit in their free plans, +while their enterprise plans may offer their clients to operate many separate units. + +The external sync unit ID is the identifier of the sync unit (project, repository, or similar) +in the external system. +For GitHub, this would be the repository, for example `cli` in `github.com/devrev/cli`. + +#### Metadata extraction + +During the metadata extraction phase, the Airdrop snap-in must provide an +`external_domain_metadata.json` file on each sync run. +This file provides a structured way of describing the external system's domain system, +including its domain entities, types, relationships, and other metadata. + +#### Data extraction + +In initial sync we need to extract all data to create a baseline while in subsequent runs only updated objects need to be extracted. + +Snap-in should consult its internal state to determine what is already extracted and what it did not + +#### Attachments extraction + +For the attachment extraction phase of the import process, the extractor has to upload each attachment to DevRev diff --git a/fern/docs/pages/airdrop/first-steps-of-development.mdx b/fern/docs/pages/airdrop/first-steps-of-development.mdx new file mode 100644 index 00000000..59d1cae3 --- /dev/null +++ b/fern/docs/pages/airdrop/first-steps-of-development.mdx @@ -0,0 +1,64 @@ +## Snap-in code structure + +A snap-in is a Node.js project with a specific structure: + +``` +- manifest.yaml +- code/ + - scripts/ + - src/ + - fixtures/ + - functions/ + - external-system/ + - extraction/ + - workers/ + - attachments-extraction.ts + - data-extraction.ts + - external-sync-units-extraction.ts + - metadata-extraction.ts + - index.ts + - install_initial_domain_mapping/ + - loading/ + - workers/ + - load-data.ts + - data-extraction.ts + - external-sync-units-extraction.ts + - metadata-extraction.ts + - index.ts + - test-runner/ + - test/ +``` + +The `manifest.yaml` file plays a critical role in a snap-in, holding key configuration details such as the snap-in's name, description, version, connection information, and functionality specifics. This file is the initial step in the development of a snap-in. + +The `code/` folder holds the implementation of the snap-in. Inside, the `functions/` subfolder includes the logic for retrieving external sync units, extracting data from and loading data to external systems, and mapping between them. + +The `scripts/` folder contains the scripts necessary for the building and deployment of the snap-in, streamlining the development and deployment processes with the help of tools like `deploy.sh` and `build.sh`. + +## Setting up the snap-in configuration + +Development of a snap-in starts with a `manifest.yaml`. +Manifest holds a snap-in configuration: snap-in name, description, version, connection information and information about implemented functionality. + +In the template change values for + +- `name` +- `description` + +## Snap-in functionalities + +This directory holds the snap-in's core logic. The logic for extracting data from the external system is found in the `extraction` folder, and the logic for loading data into the external system from DevRev is in the `loading` folder. + +The DevRev platform enables the creation of a snap-in that can either import data from an external system into DevRev or export data from DevRev to an external system. Implementing both isn't necessary. The implemented functionalities are specified in `manifest.yaml` under the `functions` key, as shown in the starter template: + +```yaml +functions: + - name: extraction + description: Extraction function for the template snap-in + - name: loading + description: Loading function for the template snap-in + - name: install_initial_domain_mapping + description: Create blueprint and install initial domain mapping +``` + +For example, removing the entry with the name loading means syncing from DevRev to the external system is not available in the UI. diff --git a/fern/docs/pages/airdrop/how-does-the-snap-in-work.mdx b/fern/docs/pages/airdrop/how-does-the-snap-in-work.mdx index 27ee0cb6..02308773 100644 --- a/fern/docs/pages/airdrop/how-does-the-snap-in-work.mdx +++ b/fern/docs/pages/airdrop/how-does-the-snap-in-work.mdx @@ -1,3 +1,25 @@ +## What happens during an airdrop + +### From the UI perspective + +1. Starting an airdrop begins with establishing a connection to the external system. The snap-in prompts for specific information needed to create this connection. Afterwards, the snap-in verifies that the connection is successful and you should see a success message. + +2. Once the connection is established, the snap-in collects information from the external system about the data unit to be exported. This phase is called _external sync units extraction_. An external sync unit is the data unit from the external system to be imported to DevRev. At this point, the UI prompts the user to select an external sync unit from the list. + +3. The snap-in then extracts data from the external system. + +4. Afterwards, the UI requests the mapping of data from the external system to DevRev. The user can decide which fields from each system should match. + +5. Once mapping is complete, the snap-in saves the data to DevRev. + +6. Finally, the snap-in streams any possible attachments from the external system to DevRev. + +The process described above is known as the initial sync. In all subsequent syncs, the first two steps are skipped. + +### In the background + +The snap-in constantly communicates with Airdrop, which manages its behavior. When an action occurs in the UI, Airdrop sends a message to the snap-in with details about the event. The snap-in performs an action based on the event type. For example, when a user starts a new airdrop for the first time, Airdrop instructs the snap-in to perform an external sync unit extraction. The snap-in carries out the task and sends back available sync units to Airdrop. + ## Snap-in code structure A snap-in is a Node.js project with a specific structure: @@ -29,84 +51,8 @@ A snap-in is a Node.js project with a specific structure: - test/ ``` -TODO: describe emitting events and invoking workers? - -#### `manifest.yaml` - -The snap-in manifest an essential part of a snap-in, as it's used to define a snap-in configuration: snap-in name, description, version, keyring types and functionality (functions). - -#### `code/` - -This folder contains the Node.js project. - -#### `scripts/` - -This folder includes scripts for deploying (`deploy.sh`) and building (`build.sh`) the snap-in using a Makefile. - -#### `functions/` - -This directory holds the snap-in's core logic. The logic for extracting data from the external system is found in the `extraction` folder, and the logic for loading data into the external system from DevRev is in the `loading` folder. - -## Snap-in functionality - -The DevRev platform enables the creation of a snap-in that can either import data from an external system into DevRev or export data from DevRev to an external system. Implementing both isn't necessary. The implemented functionalities are specified in `manifest.yaml` under the `functions` key, as shown in the starter template: - -```yaml -functions: - - name: extraction - description: Extraction function for the template snap-in - - name: loading - description: Loading function for the template snap-in - - name: install_initial_domain_mapping - description: Create blueprint and install initial domain mapping -``` - -For example, removing the entry with the name loading means syncing from DevRev to the external system is not available in the UI. - -### Extraction - -Extraction is comprised of 4 phases, which follow sequentially: - -1. external sync units extraction (only on initial import) -2. metadata extraction -3. data extraction -4. attachments extraction - -#### External sync units extraction - -In the external sync unit extraction phase, the snap-in is expected to obtain a list of external -sync units that it can extract from the external system API and send it to Airdrop in its response. - -An _external sync unit_ refers to a single unit in the external system that is being airdropped to DevRev. -In some systems, this is a project; in some it is a repository; in support systems it could be -called a brand or an organization. -What a unit of data is called and what it represents depends on the external system's domain model. -It usually combines contacts, users, work-like items, and comments into a unit of domain objects. - -Some external systems may offer a single unit in their free plans, -while their enterprise plans may offer their clients to operate many separate units. - -The external sync unit ID is the identifier of the sync unit (project, repository, or similar) -in the external system. -For GitHub, this would be the repository, for example `cli` in `github.com/devrev/cli`. - -#### Metadata extraction - -During the metadata extraction phase, the Airdrop snap-in must provide an -`external_domain_metadata.json` file on each sync run. -This file provides a structured way of describing the external system's domain system, -including its domain entities, types, relationships, and other metadata. - -#### Data extraction - -In initial sync we need to extract all data to create a baseline while in subsequent runs only updated objects need to be extracted. - -Snap-in should consult its internal state to determine what is already extracted and what it did not - -#### Attachments extraction - -For the attachment extraction phase of the import process, the extractor has to upload each attachment to DevRev +The `manifest.yaml` file is crucial in a snap-in as it contains the configuration details, including the snap-in's name, description, version, connection info, and functionality details. It serves as the starting point for snap-in development. -### Loading +The `code/` folder houses the snap-in implementation. The functions/ directory is where the core logic resides, with the extraction folder handling data extraction from external systems and the loading folder managing data loading into DevRev. -TODO +The `scripts/` folder contains scripts for building and deploying the snap-in. diff --git a/fern/docs/pages/airdrop/overview.mdx b/fern/docs/pages/airdrop/overview.mdx index 226d6353..e17ce67b 100644 --- a/fern/docs/pages/airdrop/overview.mdx +++ b/fern/docs/pages/airdrop/overview.mdx @@ -12,17 +12,17 @@ external systems. ### Sync run Airdrop extractions are done in _sync runs_. -A sync run is one execution of an import or sync. +A sync run is one execution of an airdrop or sync. ### Forward sync A _forward sync_ is a sync run from an external system to DevRev. -An extractor function in the snap-in is responsible for extracting data from the external system. +An **extractor** function in the snap-in is responsible for extracting data from the external system. ### Reverse sync A _reverse sync_ is a sync run from DevRev to an external system. -It uses a loader function, to create or update data in the external system. +It uses a **loader** function, to create or update data in the external system. ### 1-way (incremental) sync diff --git a/fern/versions/public.yml b/fern/versions/public.yml index 06f8ac43..c9d0a079 100644 --- a/fern/versions/public.yml +++ b/fern/versions/public.yml @@ -142,27 +142,30 @@ navigation: - page: "Overview" slug: overview path: ../docs/pages/airdrop/overview.mdx - - section: "Development guide" - slug: development-guide - contents: - - page: "Getting started" - slug: getting-started - path: ../docs/pages/airdrop/getting-started.mdx - - page: "How does a snap-in work" - slug: how-does-the-snap-in-work - path: ../docs/pages/airdrop/how-does-the-snap-in-work.mdx - - page: "Connecting to the external system" - slug: connecting-to-the-external-system - path: ../docs/pages/airdrop/connecting-to-the-external-system.mdx - - page: "Extracting data from external system" - slug: extracting-data - path: ../docs/pages/airdrop/extracting-data.mdx - - page: "Loading data to DevRev" - slug: loading-data - path: ../docs/pages/airdrop/loading-data.mdx - - page: "Examples of snap-ins" - slug: snap-in-examples - path: ../docs/pages/airdrop/snap-in-examples.mdx + - page: "Getting started" + slug: getting-started + path: ../docs/pages/airdrop/getting-started.mdx + - page: "How does a snap-in work" + slug: how-does-the-snap-in-work + path: ../docs/pages/airdrop/how-does-the-snap-in-work.mdx + - page: "Connecting to the external system" + slug: connecting-to-the-external-system + path: ../docs/pages/airdrop/connecting-to-the-external-system.mdx + - page: "Extracting data from external system" + slug: extracting-data + path: ../docs/pages/airdrop/extracting-data.mdx + - page: "Loading data to DevRev" + slug: loading-data + path: ../docs/pages/airdrop/loading-data.mdx + - page: "Examples of snap-ins" + slug: snap-in-examples + path: ../docs/pages/airdrop/snap-in-examples.mdx + - page: "Publish to the marketplace" + slug: publish-to-marketplace + path: ../docs/pages/airdrop/publish-to-marketplace.mdx + - page: "Common issues - FAQ" + slug: faq + path: ../docs/pages/airdrop/faq.mdx - page: "Extraction phases" slug: extraction-phases hidden: true @@ -191,12 +194,6 @@ navigation: slug: data-attachments-deletion hidden: true path: ../docs/pages/airdrop/data-attachments-deletion.mdx - - page: "Publish to the marketplace" - slug: publish-to-marketplace - path: ../docs/pages/airdrop/publish-to-marketplace.mdx - - page: "Common issues - FAQ" - slug: faq - path: ../docs/pages/airdrop/faq.mdx - section: References slug: references contents: From dc4624f8ed6a87567641c678b39c265049a215ee Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Patricija=20Bre=C4=8Dko?= Date: Wed, 26 Mar 2025 11:29:37 +0100 Subject: [PATCH 15/27] Add first steps of development to navigation --- .../airdrop/how-does-the-snap-in-work.mdx | 37 ------------------- fern/versions/public.yml | 3 ++ 2 files changed, 3 insertions(+), 37 deletions(-) diff --git a/fern/docs/pages/airdrop/how-does-the-snap-in-work.mdx b/fern/docs/pages/airdrop/how-does-the-snap-in-work.mdx index 02308773..7827225e 100644 --- a/fern/docs/pages/airdrop/how-does-the-snap-in-work.mdx +++ b/fern/docs/pages/airdrop/how-does-the-snap-in-work.mdx @@ -19,40 +19,3 @@ The process described above is known as the initial sync. In all subsequent sync ### In the background The snap-in constantly communicates with Airdrop, which manages its behavior. When an action occurs in the UI, Airdrop sends a message to the snap-in with details about the event. The snap-in performs an action based on the event type. For example, when a user starts a new airdrop for the first time, Airdrop instructs the snap-in to perform an external sync unit extraction. The snap-in carries out the task and sends back available sync units to Airdrop. - -## Snap-in code structure - -A snap-in is a Node.js project with a specific structure: - -``` -- manifest.yaml -- code/ - - scripts/ - - src/ - - fixtures/ - - functions/ - - external-system/ - - extraction/ - - workers/ - - attachments-extraction.ts - - data-extraction.ts - - external-sync-units-extraction.ts - - metadata-extraction.ts - - index.ts - - install_initial_domain_mapping/ - - loading/ - - workers/ - - load-data.ts - - data-extraction.ts - - external-sync-units-extraction.ts - - metadata-extraction.ts - - index.ts - - test-runner/ - - test/ -``` - -The `manifest.yaml` file is crucial in a snap-in as it contains the configuration details, including the snap-in's name, description, version, connection info, and functionality details. It serves as the starting point for snap-in development. - -The `code/` folder houses the snap-in implementation. The functions/ directory is where the core logic resides, with the extraction folder handling data extraction from external systems and the loading folder managing data loading into DevRev. - -The `scripts/` folder contains scripts for building and deploying the snap-in. diff --git a/fern/versions/public.yml b/fern/versions/public.yml index c9d0a079..a0bd0fad 100644 --- a/fern/versions/public.yml +++ b/fern/versions/public.yml @@ -148,6 +148,9 @@ navigation: - page: "How does a snap-in work" slug: how-does-the-snap-in-work path: ../docs/pages/airdrop/how-does-the-snap-in-work.mdx + - page: "First steps of development" + slug: first-steps-of-development + path: ../docs/pages/airdrop/first-steps-of-development.mdx - page: "Connecting to the external system" slug: connecting-to-the-external-system path: ../docs/pages/airdrop/connecting-to-the-external-system.mdx From 9b545f398fb908b4ff5c7358a0d06bab5f6684b0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Patricija=20Bre=C4=8Dko?= Date: Wed, 26 Mar 2025 12:01:48 +0100 Subject: [PATCH 16/27] Move some things around --- .../pages/airdrop/deploy-to-organization.mdx | 27 ++++++++ fern/docs/pages/airdrop/extracting-data.mdx | 2 +- .../airdrop/first-steps-of-development.mdx | 64 ------------------- fern/docs/pages/airdrop/getting-started.mdx | 47 ++++++++------ .../airdrop/how-does-the-snap-in-work.mdx | 34 +++++++++- fern/docs/pages/airdrop/overview.mdx | 32 ---------- .../pages/airdrop/publish-to-marketplace.mdx | 8 +-- .../pages/airdrop/snap-in-configuration.mdx | 27 ++++++++ fern/versions/public.yml | 11 ++-- 9 files changed, 128 insertions(+), 124 deletions(-) create mode 100644 fern/docs/pages/airdrop/deploy-to-organization.mdx delete mode 100644 fern/docs/pages/airdrop/first-steps-of-development.mdx create mode 100644 fern/docs/pages/airdrop/snap-in-configuration.mdx diff --git a/fern/docs/pages/airdrop/deploy-to-organization.mdx b/fern/docs/pages/airdrop/deploy-to-organization.mdx new file mode 100644 index 00000000..cdbfe3e2 --- /dev/null +++ b/fern/docs/pages/airdrop/deploy-to-organization.mdx @@ -0,0 +1,27 @@ +Once you're ready to test your snap-in in a production environment, you can deploy the snap-in to your organization. + +Follow these steps: + +1. Copy `.env.example` to a new file named `.env` and fill in the required variables. +2. Deploy a draft version of your snap-in to your organization by using `make deploy`. +3. Install the snap-in in your DevRev by going to **Settings** > **Snap-ins** > **Install snap-in**. +4. Set up the connection under **Settings** > **Airdrops** > **Connections**. +5. Create an import at **Settings** > **Airdrops** > **Airdrop**. + +This step is also a prerequisite for publishing the snap-in on the DevRev marketplace. + +### Observability + +To observe logs from your snap-in in your development environment: + +``` +devrev snap_in_package logs | jq +``` + +To open logs in your favorite editor: + +``` +devrev snap_in_package logs | code - +``` + +For more information, refer to [Debugging](/snapin-development/debugging). diff --git a/fern/docs/pages/airdrop/extracting-data.mdx b/fern/docs/pages/airdrop/extracting-data.mdx index 0a4763c0..af9c2a85 100644 --- a/fern/docs/pages/airdrop/extracting-data.mdx +++ b/fern/docs/pages/airdrop/extracting-data.mdx @@ -42,4 +42,4 @@ Snap-in should consult its internal state to determine what is already extracted #### Attachments extraction -For the attachment extraction phase of the import process, the extractor has to upload each attachment to DevRev +For the attachment extraction phase of the import process, the extractor has to upload each attachment to DevRev and associate it with the data object it is attached to. diff --git a/fern/docs/pages/airdrop/first-steps-of-development.mdx b/fern/docs/pages/airdrop/first-steps-of-development.mdx deleted file mode 100644 index 59d1cae3..00000000 --- a/fern/docs/pages/airdrop/first-steps-of-development.mdx +++ /dev/null @@ -1,64 +0,0 @@ -## Snap-in code structure - -A snap-in is a Node.js project with a specific structure: - -``` -- manifest.yaml -- code/ - - scripts/ - - src/ - - fixtures/ - - functions/ - - external-system/ - - extraction/ - - workers/ - - attachments-extraction.ts - - data-extraction.ts - - external-sync-units-extraction.ts - - metadata-extraction.ts - - index.ts - - install_initial_domain_mapping/ - - loading/ - - workers/ - - load-data.ts - - data-extraction.ts - - external-sync-units-extraction.ts - - metadata-extraction.ts - - index.ts - - test-runner/ - - test/ -``` - -The `manifest.yaml` file plays a critical role in a snap-in, holding key configuration details such as the snap-in's name, description, version, connection information, and functionality specifics. This file is the initial step in the development of a snap-in. - -The `code/` folder holds the implementation of the snap-in. Inside, the `functions/` subfolder includes the logic for retrieving external sync units, extracting data from and loading data to external systems, and mapping between them. - -The `scripts/` folder contains the scripts necessary for the building and deployment of the snap-in, streamlining the development and deployment processes with the help of tools like `deploy.sh` and `build.sh`. - -## Setting up the snap-in configuration - -Development of a snap-in starts with a `manifest.yaml`. -Manifest holds a snap-in configuration: snap-in name, description, version, connection information and information about implemented functionality. - -In the template change values for - -- `name` -- `description` - -## Snap-in functionalities - -This directory holds the snap-in's core logic. The logic for extracting data from the external system is found in the `extraction` folder, and the logic for loading data into the external system from DevRev is in the `loading` folder. - -The DevRev platform enables the creation of a snap-in that can either import data from an external system into DevRev or export data from DevRev to an external system. Implementing both isn't necessary. The implemented functionalities are specified in `manifest.yaml` under the `functions` key, as shown in the starter template: - -```yaml -functions: - - name: extraction - description: Extraction function for the template snap-in - - name: loading - description: Loading function for the template snap-in - - name: install_initial_domain_mapping - description: Create blueprint and install initial domain mapping -``` - -For example, removing the entry with the name loading means syncing from DevRev to the external system is not available in the UI. diff --git a/fern/docs/pages/airdrop/getting-started.mdx b/fern/docs/pages/airdrop/getting-started.mdx index 848136c2..bffeb455 100644 --- a/fern/docs/pages/airdrop/getting-started.mdx +++ b/fern/docs/pages/airdrop/getting-started.mdx @@ -62,28 +62,39 @@ DevRev provides a template with example code to implement an Airdrop as a Servic ``` 9. Start the import (**Airdrops** -> **Start Airdrop** -> **Your snap-in**). -## Publishing the snap-in +## Snap-in code structure -Once you're ready to launch your snap-in in a production environment, follow these steps: - -1. Copy `.env.example` to a new file named `.env` and fill in the required variables. -2. Deploy a draft version of your snap-in to your organization by using `make deploy`. -3. Install the snap-in in your DevRev by going to **Settings** > **Snap-ins** > **Install snap-in**. -4. Set up the connection under **Settings** > **Airdrops** > **Connections**. -5. Create an import at **Settings** > **Airdrops** > **Airdrop**. - -### Observability - -To observe logs from your snap-in in your development environment: +A snap-in is a Node.js project with a specific file structure: ``` -devrev snap_in_package logs | jq +- manifest.yaml +- code/ + - scripts/ + - src/ + - fixtures/ + - functions/ + - external-system/ + - extraction/ + - workers/ + - attachments-extraction.ts + - data-extraction.ts + - external-sync-units-extraction.ts + - metadata-extraction.ts + - index.ts + - install_initial_domain_mapping/ + - loading/ + - workers/ + - load-data.ts + - data-extraction.ts + - external-sync-units-extraction.ts + - metadata-extraction.ts + - index.ts + - test-runner/ + - test/ ``` -To open logs in your favorite editor: +The `manifest.yaml` file plays a critical role in a snap-in, holding key configuration details such as the snap-in's name, description, version, connection information, and functionality specifics. This file is the initial step in the development of a snap-in. -``` -devrev snap_in_package logs | code - -``` +The `code/` folder holds the implementation of the snap-in. Inside, the `functions/` subfolder includes the logic for retrieving external sync units, extracting data from and loading data to external systems, and mapping between them. -For more information, refer to [Debugging](/snapin-development/debugging). +The `scripts/` folder contains the scripts necessary for the building and deployment of the snap-in, streamlining the development and deployment processes with the help of tools like `deploy.sh` and `build.sh`. diff --git a/fern/docs/pages/airdrop/how-does-the-snap-in-work.mdx b/fern/docs/pages/airdrop/how-does-the-snap-in-work.mdx index 7827225e..22ed3f46 100644 --- a/fern/docs/pages/airdrop/how-does-the-snap-in-work.mdx +++ b/fern/docs/pages/airdrop/how-does-the-snap-in-work.mdx @@ -1,4 +1,4 @@ -## What happens during an airdrop +In order to start an airdrop, go to UI and click **Airdrops** -> **Start Airdrop** -> **Your snap-in**. ### From the UI perspective @@ -19,3 +19,35 @@ The process described above is known as the initial sync. In all subsequent sync ### In the background The snap-in constantly communicates with Airdrop, which manages its behavior. When an action occurs in the UI, Airdrop sends a message to the snap-in with details about the event. The snap-in performs an action based on the event type. For example, when a user starts a new airdrop for the first time, Airdrop instructs the snap-in to perform an external sync unit extraction. The snap-in carries out the task and sends back available sync units to Airdrop. + +## Terminology + +### Sync run + +Airdrop extractions are done in _sync runs_. +A sync run is one execution of an airdrop or sync. + +### Forward sync + +A _forward sync_ is a sync run from an external system to DevRev. +An **extractor** function in the snap-in is responsible for extracting data from the external system. + +### Reverse sync + +A _reverse sync_ is a sync run from DevRev to an external system. +It uses a **loader** function, to create or update data in the external system. + +### 1-way (incremental) sync + +A _1-way sync_ (or _incremental sync_) refers to any extraction after the initial import has been successfully completed. +An extractor extracts data that was created or updated in the external system after the start +of the latest successful forward sync, including any changes that occurred during the forward sync, +but were not picked up by it. + +A snap-in must consult its state to get information on when the last successful forward sync started. +Airdrop snap-ins must maintain their own states that persists between phases in a sync run, +as well as between sync runs. + +A 1-way sync extracts only the domain objects updated or created since the previous successful sync run. + +TODO: define the sync unit and how it relates to external project/sync unit, ... diff --git a/fern/docs/pages/airdrop/overview.mdx b/fern/docs/pages/airdrop/overview.mdx index e17ce67b..895377ef 100644 --- a/fern/docs/pages/airdrop/overview.mdx +++ b/fern/docs/pages/airdrop/overview.mdx @@ -7,36 +7,4 @@ DevRev’s Airdrop functionality. It enables developers to create external workers (extractors and loaders) to bring data from various external systems. -## Terminology - -### Sync run - -Airdrop extractions are done in _sync runs_. -A sync run is one execution of an airdrop or sync. - -### Forward sync - -A _forward sync_ is a sync run from an external system to DevRev. -An **extractor** function in the snap-in is responsible for extracting data from the external system. - -### Reverse sync - -A _reverse sync_ is a sync run from DevRev to an external system. -It uses a **loader** function, to create or update data in the external system. - -### 1-way (incremental) sync - -A _1-way sync_ (or _incremental sync_) refers to any extraction after the initial import has been successfully completed. -An extractor extracts data that was created or updated in the external system after the start -of the latest successful forward sync, including any changes that occurred during the forward sync, -but were not picked up by it. - -A snap-in must consult its state to get information on when the last successful forward sync started. -Airdrop snap-ins must maintain their own states that persists between phases in a sync run, -as well as between sync runs. - -A 1-way sync extracts only the domain objects updated or created since the previous successful sync run. - -TODO: define the sync unit and how it relates to external project/sync unit, ... - You can learn more about Airdrop in the [general documentation](https://docs.devrev.ai/import#airdrop-features). diff --git a/fern/docs/pages/airdrop/publish-to-marketplace.mdx b/fern/docs/pages/airdrop/publish-to-marketplace.mdx index ca0f731c..2844aae6 100644 --- a/fern/docs/pages/airdrop/publish-to-marketplace.mdx +++ b/fern/docs/pages/airdrop/publish-to-marketplace.mdx @@ -2,7 +2,7 @@ When your snap-in is ready for use by you or other organizations, publish it to ### 1. Create marketplace listing -When creating a new marketplace listing, an initial marketplace submission must be created with all the mandatory properties. To create a new listing, run the following command, which will start a wizard to guide you through the process: +When creating a new marketplace listing, an initial marketplace submission must be created with all the mandatory properties. To create a new listing, run the following command, which starts a wizard to guide you through the process: ```bash devrev marketplace_submissions create @@ -10,13 +10,13 @@ devrev marketplace_submissions create ### 2. Submit marketplace listing for review -The newly created marketplace submission is in the Draft state. In order to publish it, we first have to transition it to Waiting for review state. This can be done by running the following command, which will start a wizard to guide you through the process: +The newly created marketplace submission is in the Draft state. In order to publish it, we first have to transition it to Waiting for review state. This can be done by running the following command, which starts a wizard to guide you through the process: ```bash devrev marketplace_submissions transition ``` -Once the submission is transitioned to the Waiting for review state, it needs to be approved by a marketplace admin. While in review, the state of the submission is In review. Once it is reviewed, the state will be changed to either Approved or Rejected. +Once the submission is transitioned to the Waiting for review state, it needs to be approved by a marketplace admin. While in review, the state of the submission is In review. Once it is reviewed, the state changes to either Approved or Rejected. Submissions whose state is _In review_ or _Approved_ cannot be modified. @@ -26,7 +26,7 @@ If the submission was rejected, you can transition it back to Draft and modify i ### 3. Publish marketplace submission -If the submission was approved, you can publish it using the following command, which will start a wizard to guide you through the process: +If the submission is approved, you can publish it using the following command, which starts a wizard to guide you through the process: ```bash devrev marketplace_items publish diff --git a/fern/docs/pages/airdrop/snap-in-configuration.mdx b/fern/docs/pages/airdrop/snap-in-configuration.mdx new file mode 100644 index 00000000..1964a35e --- /dev/null +++ b/fern/docs/pages/airdrop/snap-in-configuration.mdx @@ -0,0 +1,27 @@ +## Setting up the snap-in configuration + +Development of a snap-in starts with a `manifest.yaml`. +Manifest holds a snap-in configuration: snap-in name, description, version, connection information and information about implemented functionality. + +In the template change values for + +- `name` +- `description` + +## Snap-in functionalities + +This directory holds the snap-in's core logic. The logic for extracting data from the external system is found in the `extraction` folder, and the logic for loading data into the external system from DevRev is in the `loading` folder. + +The DevRev platform enables the creation of a snap-in that can either import data from an external system into DevRev or export data from DevRev to an external system. Implementing both isn't necessary. The implemented functionalities are specified in `manifest.yaml` under the `functions` key, as shown in the starter template: + +```yaml +functions: + - name: extraction + description: Extraction function for the template snap-in + - name: loading + description: Loading function for the template snap-in + - name: install_initial_domain_mapping + description: Create blueprint and install initial domain mapping +``` + +For example, removing the entry with the name loading means syncing from DevRev to the external system is not available in the UI. diff --git a/fern/versions/public.yml b/fern/versions/public.yml index a0bd0fad..f306fa97 100644 --- a/fern/versions/public.yml +++ b/fern/versions/public.yml @@ -145,12 +145,12 @@ navigation: - page: "Getting started" slug: getting-started path: ../docs/pages/airdrop/getting-started.mdx - - page: "How does a snap-in work" + - page: "Start an airdrop" slug: how-does-the-snap-in-work path: ../docs/pages/airdrop/how-does-the-snap-in-work.mdx - - page: "First steps of development" - slug: first-steps-of-development - path: ../docs/pages/airdrop/first-steps-of-development.mdx + - page: "Snap-in configuration" + slug: snap-in-configuration + path: ../docs/pages/airdrop/snap-in-configuration.mdx - page: "Connecting to the external system" slug: connecting-to-the-external-system path: ../docs/pages/airdrop/connecting-to-the-external-system.mdx @@ -163,6 +163,9 @@ navigation: - page: "Examples of snap-ins" slug: snap-in-examples path: ../docs/pages/airdrop/snap-in-examples.mdx + - page: "Deploy to organization" + slug: deploy-to-organization + path: ../docs/pages/airdrop/deploy-to-organization.mdx - page: "Publish to the marketplace" slug: publish-to-marketplace path: ../docs/pages/airdrop/publish-to-marketplace.mdx From 2d8083ae8cdd1595895fa57c4451e458a84bc8b9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Patricija=20Bre=C4=8Dko?= Date: Wed, 26 Mar 2025 13:34:15 +0100 Subject: [PATCH 17/27] Snap-in configuration --- .../connecting-to-the-external-system.mdx | 2 -- fern/docs/pages/airdrop/getting-started.mdx | 5 ++- .../airdrop/how-does-the-snap-in-work.mdx | 25 ++++++------- .../pages/airdrop/snap-in-configuration.mdx | 35 +++++++++++++++---- 4 files changed, 41 insertions(+), 26 deletions(-) diff --git a/fern/docs/pages/airdrop/connecting-to-the-external-system.mdx b/fern/docs/pages/airdrop/connecting-to-the-external-system.mdx index e2d776b6..b2c059ef 100644 --- a/fern/docs/pages/airdrop/connecting-to-the-external-system.mdx +++ b/fern/docs/pages/airdrop/connecting-to-the-external-system.mdx @@ -1,5 +1,3 @@ -## Keyrings - _Keyrings_ are a collection of authentication information, used by a snap-in to authenticate to the external system in API calls. This can include a key (for example, a PAT token or API key), its type, the organization ID for which a key is valid, and in some cases the organization name. Keyrings provide a secure way to store and manage credentials within your DevRev snap-in. diff --git a/fern/docs/pages/airdrop/getting-started.mdx b/fern/docs/pages/airdrop/getting-started.mdx index bffeb455..e22668a9 100644 --- a/fern/docs/pages/airdrop/getting-started.mdx +++ b/fern/docs/pages/airdrop/getting-started.mdx @@ -60,11 +60,10 @@ DevRev provides a template with example code to implement an Airdrop as a Servic ```bash devrev snap_in activate ``` -9. Start the import (**Airdrops** -> **Start Airdrop** -> **Your snap-in**). -## Snap-in code structure +### Snap-in code structure -A snap-in is a Node.js project with a specific file structure: +If you check the template, you can see that a snap-in is a Node.js project with a specific file structure: ``` - manifest.yaml diff --git a/fern/docs/pages/airdrop/how-does-the-snap-in-work.mdx b/fern/docs/pages/airdrop/how-does-the-snap-in-work.mdx index 22ed3f46..165ca357 100644 --- a/fern/docs/pages/airdrop/how-does-the-snap-in-work.mdx +++ b/fern/docs/pages/airdrop/how-does-the-snap-in-work.mdx @@ -1,6 +1,7 @@ -In order to start an airdrop, go to UI and click **Airdrops** -> **Start Airdrop** -> **Your snap-in**. +Now that you have a running snap-in, you can start an airdrop. +Go to UI and click **Airdrops** -> **Start Airdrop** -> **Your snap-in**. -### From the UI perspective +### What happens in the UI 1. Starting an airdrop begins with establishing a connection to the external system. The snap-in prompts for specific information needed to create this connection. Afterwards, the snap-in verifies that the connection is successful and you should see a success message. @@ -14,9 +15,9 @@ In order to start an airdrop, go to UI and click **Airdrops** -> **Start Airdrop 6. Finally, the snap-in streams any possible attachments from the external system to DevRev. -The process described above is known as the initial sync. In all subsequent syncs, the first two steps are skipped. +The process described above is called the **initial sync**. In all subsequent syncs, the first two steps are skipped. -### In the background +### What happens in the background The snap-in constantly communicates with Airdrop, which manages its behavior. When an action occurs in the UI, Airdrop sends a message to the snap-in with details about the event. The snap-in performs an action based on the event type. For example, when a user starts a new airdrop for the first time, Airdrop instructs the snap-in to perform an external sync unit extraction. The snap-in carries out the task and sends back available sync units to Airdrop. @@ -37,17 +38,13 @@ An **extractor** function in the snap-in is responsible for extracting data from A _reverse sync_ is a sync run from DevRev to an external system. It uses a **loader** function, to create or update data in the external system. +### Initial sync + +It is the first forward sync to run. In initial sync we need to extract all data to create a baseline (while in incremental runs only updated objects need to be extracted). + ### 1-way (incremental) sync -A _1-way sync_ (or _incremental sync_) refers to any extraction after the initial import has been successfully completed. -An extractor extracts data that was created or updated in the external system after the start +A _1-way sync_ (or _incremental sync_) refers to any extraction after the initial sync run has been successfully completed. +In the incremental sync an extractor extracts data that was created or updated in the external system after the start of the latest successful forward sync, including any changes that occurred during the forward sync, but were not picked up by it. - -A snap-in must consult its state to get information on when the last successful forward sync started. -Airdrop snap-ins must maintain their own states that persists between phases in a sync run, -as well as between sync runs. - -A 1-way sync extracts only the domain objects updated or created since the previous successful sync run. - -TODO: define the sync unit and how it relates to external project/sync unit, ... diff --git a/fern/docs/pages/airdrop/snap-in-configuration.mdx b/fern/docs/pages/airdrop/snap-in-configuration.mdx index 1964a35e..e0de7405 100644 --- a/fern/docs/pages/airdrop/snap-in-configuration.mdx +++ b/fern/docs/pages/airdrop/snap-in-configuration.mdx @@ -1,18 +1,18 @@ -## Setting up the snap-in configuration - Development of a snap-in starts with a `manifest.yaml`. Manifest holds a snap-in configuration: snap-in name, description, version, connection information and information about implemented functionality. -In the template change values for +Begin by setting the values in the template for - `name` - `description` -## Snap-in functionalities +### Set snap-in functionalities + +The DevRev platform enables the creation of a snap-in that can import data from an external system into DevRev (called _extraction_) and export data from DevRev to an external system (called _loading_). Implementation of extration is necessary, while of loading isn't. -This directory holds the snap-in's core logic. The logic for extracting data from the external system is found in the `extraction` folder, and the logic for loading data into the external system from DevRev is in the `loading` folder. +To specify the implemented functionalities, first review the `functions` section of `manifest.yaml`. -The DevRev platform enables the creation of a snap-in that can either import data from an external system into DevRev or export data from DevRev to an external system. Implementing both isn't necessary. The implemented functionalities are specified in `manifest.yaml` under the `functions` key, as shown in the starter template: +In case you're not going to implement loading, remove it from the list. ```yaml functions: @@ -24,4 +24,25 @@ functions: description: Create blueprint and install initial domain mapping ``` -For example, removing the entry with the name loading means syncing from DevRev to the external system is not available in the UI. +Removing the entry with the name `loading` will also remove this option from the UI. + +Next, check the `imports` section. + +```yaml +imports: + - slug: adaas-template + display_name: ADaaS Connector Template + description: ADaaS Connector Template + extractor_function: extraction + loader_function: loading + allowed_connection_types: + - example-connection +``` + +Change `slug`, `display_name` and `description` according to your snap-in. + +Remove `loader_function` in case you're not using it. + +Make sure the names for `extractor_function` and `loader_function` match your settings in the `functions` section. + +When you are done you can continue with the next section that focuses on `keyring_types`. From fe2cad993855816fca07ca32b42a20010eb73db2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Patricija=20Bre=C4=8Dko?= Date: Thu, 27 Mar 2025 09:23:38 +0100 Subject: [PATCH 18/27] Split data extraction into sections --- .../external-sync-units-extraction.mdx | 74 ++++++++++--------- .../airdrop/extracting-data-and-metadata.mdx | 43 +++++++++++ fern/docs/pages/airdrop/extracting-data.mdx | 54 ++++---------- .../pages/airdrop/snap-in-configuration.mdx | 14 ++-- fern/versions/public.yml | 13 +++- 5 files changed, 116 insertions(+), 82 deletions(-) create mode 100644 fern/docs/pages/airdrop/extracting-data-and-metadata.mdx diff --git a/fern/docs/pages/airdrop/external-sync-units-extraction.mdx b/fern/docs/pages/airdrop/external-sync-units-extraction.mdx index 97e34f66..891f8ec0 100644 --- a/fern/docs/pages/airdrop/external-sync-units-extraction.mdx +++ b/fern/docs/pages/airdrop/external-sync-units-extraction.mdx @@ -1,52 +1,60 @@ -In the external sync unit extraction phase, the extractor is expected to obtain a list of external -sync units that it can extract with the provided credentials and send it to Airdrop in its response. - An _external sync unit_ refers to a single unit in the external system that is being airdropped to DevRev. In some systems, this is a project; in some it is a repository; in support systems it could be called a brand or an organization. What a unit of data is called and what it represents depends on the external system's domain model. It usually combines contacts, users, work-like items, and comments into a unit of domain objects. -Some external systems may offer a single unit in their free plans, -while their enterprise plans may offer their clients to operate many separate units. - -The external sync unit ID is the identifier of the sync unit (project, repository, or similar) -in the external system. -For GitHub, this would be the repository, for example `cli` in `github.com/devrev/cli`. - -## Triggering event +In the external sync unit extraction phase, the snap-in is expected to obtain a list of external +sync units that it can extract from the external system API and send it to Airdrop in its response. External sync unit extraction is executed only during the initial import. -It extracts external sync units available in the external system, so that the end user can choose -which external sync unit should be airdropped during the creation of an **Import** in the DevRev App. - -Airdrop initiates the external sync unit extraction phase by starting the worker with a message -with an event of type `EXTRACTION_EXTERNAL_SYNC_UNITS_START`. -The snap-in must respond to Airdrop with a message with an event of type -`EXTRACTION_EXTERNAL_SYNC_UNITS_DONE`, which contains a list of external sync units as a payload, -or `EXTRACTION_EXTERNAL_SYNC_UNITS_ERROR` in case of an error. +### Implementation -## Response from the snap-in +This phase should be implemented in the [`external-sync-units-extraction.ts`](https://github.com/devrev/adaas-template/blob/main/code/src/functions/extraction/workers/external-sync-units-extraction.ts) file. -The snap-in provides the list of external sync units in the provided event message +The snap-in should emit the list of external sync units in the provided event message `event_data.external_sync_units` containing the following fields: + - `id`: The unique identifier in the external system. - `name`: The human-readable name in the external system. - `description`: The short description if the external system provides it. - `item_count`: The number of items (issues, tickets, comments or others) in the external system. -Item count should be provided if it can be obtained in a lightweight manner, such as by calling an API endpoint. -If there is no such way to get it (for example, if the items would need to be extracted to count them), -then the item count should be `-1` to avoid blocking the import with long-running queries. + Item count should be provided if it can be obtained in a lightweight manner, such as by calling an API endpoint. + If there is no such way to get it (for example, if the items would need to be extracted to count them), + then the item count should be `-1` to avoid blocking the import with long-running queries. -Example: -```json -[ +```typescript +const externalSyncUnits: ExternalSyncUnit[] = [ { - "id": "a-microservice-repository", - "name": "A Microservice Repository", - "description": "Our greatest microservice repo", - "item_count": 232 - } -] + id: "devrev", + name: "devrev", + description: "Demo external sync unit", + item_count: 100, + }, +]; +``` + +The snap-in must respond to Airdrop with a message, which contains a list of external sync units as a payload: + +```typescript +await adapter.emit(ExtractorEventType.ExtractionExternalSyncUnitsDone, { + external_sync_units: externalSyncUnits, +}); +``` + +or an error: + +```typescript +await adapter.emit(ExtractorEventType.ExtractionExternalSyncUnitsError, { + error: { + message: "Failed to extract external sync units. Lambda timeout.", + }, +}); ``` + +### UI + +Start a new airdrop in the DevRev App. + +If external sync units extraction is successful, you should be prompted to choose an external sync unit from the list. diff --git a/fern/docs/pages/airdrop/extracting-data-and-metadata.mdx b/fern/docs/pages/airdrop/extracting-data-and-metadata.mdx new file mode 100644 index 00000000..0b8c99ca --- /dev/null +++ b/fern/docs/pages/airdrop/extracting-data-and-metadata.mdx @@ -0,0 +1,43 @@ +### Metadata and data extraction + + + It is recommended that you develop the metadata and data extraction phases + iteratively. The idea is that you first try to get one type of object with a + few simple fields, then continue to extend functionality and test the + implementation throughout the process. + + +The idea for metadata extraction phase is that the snap-in provides an +`external_domain_metadata.json` file to Airdrop. + +```typescript +import externalDomainMetadata from "../../external-system/external_domain_metadata.json"; + +// ... + +await adapter + .getRepo("external_domain_metadata") + ?.push([externalDomainMetadata]); +await adapter.emit(ExtractorEventType.ExtractionMetadataDone); +``` + +Metadata extraction should be implemented in the [metadata-extraction.ts](https://github.com/devrev/adaas-template/blob/main/code/src/functions/extraction/workers/metadata-extraction.ts) file. + +#### Preparing `external_domain_metadata.json` + +This `external_domain_metadata.json` file provides a structured way of describing the external system's domain system, +including its domain entities, types, relationships, and other metadata. + +Check the [example](https://github.com/devrev/adaas-template/blob/main/code/src/functions/external-system/external_domain_metadata.json) from the starter template. + +The `record_types` object should contain all the objects that will be imported from the external system and specifications for their fields. + +#### Data extraction + +In initial sync we need to extract all data to create a baseline while in subsequent runs only updated objects need to be extracted. + +Snap-in should consult its internal state to determine what is already extracted and what it did not + +#### Attachments extraction + +For the attachment extraction phase of the import process, the extractor has to upload each attachment to DevRev and associate it with the data object it is attached to. diff --git a/fern/docs/pages/airdrop/extracting-data.mdx b/fern/docs/pages/airdrop/extracting-data.mdx index af9c2a85..f4232f41 100644 --- a/fern/docs/pages/airdrop/extracting-data.mdx +++ b/fern/docs/pages/airdrop/extracting-data.mdx @@ -1,45 +1,19 @@ -## Snap-in functionality +Data extraction involves four sequential phases: -### Extraction +1. external sync units extraction (only on initial import), +2. metadata extraction, +3. data extraction, +4. attachments extraction. -Extraction is comprised of 4 phases, which follow sequentially: +### Before you start implementation -1. external sync units extraction (only on initial import) -2. metadata extraction -3. data extraction -4. attachments extraction +It is recommended you consider the following points before you start the implementation: -#### External sync units extraction +1. Examine the available objects in both the external system and DevRev and decide how they should correspond to each other. +2. Identify required fields in DevRev and available fields in the external system API. Plan mappings accordingly, considering how to handle required fields in one system that are optional in the other. +3. Check the external system API too see what are options and limitations. -In the external sync unit extraction phase, the snap-in is expected to obtain a list of external -sync units that it can extract from the external system API and send it to Airdrop in its response. - -An _external sync unit_ refers to a single unit in the external system that is being airdropped to DevRev. -In some systems, this is a project; in some it is a repository; in support systems it could be -called a brand or an organization. -What a unit of data is called and what it represents depends on the external system's domain model. -It usually combines contacts, users, work-like items, and comments into a unit of domain objects. - -Some external systems may offer a single unit in their free plans, -while their enterprise plans may offer their clients to operate many separate units. - -The external sync unit ID is the identifier of the sync unit (project, repository, or similar) -in the external system. -For GitHub, this would be the repository, for example `cli` in `github.com/devrev/cli`. - -#### Metadata extraction - -During the metadata extraction phase, the Airdrop snap-in must provide an -`external_domain_metadata.json` file on each sync run. -This file provides a structured way of describing the external system's domain system, -including its domain entities, types, relationships, and other metadata. - -#### Data extraction - -In initial sync we need to extract all data to create a baseline while in subsequent runs only updated objects need to be extracted. - -Snap-in should consult its internal state to determine what is already extracted and what it did not - -#### Attachments extraction - -For the attachment extraction phase of the import process, the extractor has to upload each attachment to DevRev and associate it with the data object it is attached to. + + Full specification of objects available in DevRev is + [here](https://github.com/devrev/airdrop-recipe-manager/blob/main/v2/devrev_metadata/devrev_metadata_docs.md) + diff --git a/fern/docs/pages/airdrop/snap-in-configuration.mdx b/fern/docs/pages/airdrop/snap-in-configuration.mdx index e0de7405..57d02640 100644 --- a/fern/docs/pages/airdrop/snap-in-configuration.mdx +++ b/fern/docs/pages/airdrop/snap-in-configuration.mdx @@ -1,10 +1,12 @@ Development of a snap-in starts with a `manifest.yaml`. Manifest holds a snap-in configuration: snap-in name, description, version, connection information and information about implemented functionality. -Begin by setting the values in the template for +Begin by setting the values in the template for `name` and `description`. -- `name` -- `description` +```yaml +name: ADaaS Template +description: A template for creating a new ADaaS snap-in +``` ### Set snap-in functionalities @@ -12,8 +14,6 @@ The DevRev platform enables the creation of a snap-in that can import data from To specify the implemented functionalities, first review the `functions` section of `manifest.yaml`. -In case you're not going to implement loading, remove it from the list. - ```yaml functions: - name: extraction @@ -24,7 +24,7 @@ functions: description: Create blueprint and install initial domain mapping ``` -Removing the entry with the name `loading` will also remove this option from the UI. +In case you're not going to implement loading, remove it from the list. This will also remove this option from the UI. Next, check the `imports` section. @@ -45,4 +45,4 @@ Remove `loader_function` in case you're not using it. Make sure the names for `extractor_function` and `loader_function` match your settings in the `functions` section. -When you are done you can continue with the next section that focuses on `keyring_types`. +When you are done you can continue with the next section that focuses on establishing a connection to the external system using `keyring_types`. diff --git a/fern/versions/public.yml b/fern/versions/public.yml index f306fa97..1651eca4 100644 --- a/fern/versions/public.yml +++ b/fern/versions/public.yml @@ -154,9 +154,18 @@ navigation: - page: "Connecting to the external system" slug: connecting-to-the-external-system path: ../docs/pages/airdrop/connecting-to-the-external-system.mdx - - page: "Extracting data from external system" + - section: Extracting data from external system slug: extracting-data - path: ../docs/pages/airdrop/extracting-data.mdx + contents: + - page: "Getting started with extraction" + slug: getting-started + path: ../docs/pages/airdrop/extracting-data.mdx + - page: "External sync units extraction" + slug: external-sync-units-extraction + path: ../docs/pages/airdrop/external-sync-units-extraction.mdx + - page: "Extracting metadata and data" + slug: extracting-data + path: ../docs/pages/airdrop/extracting-data-and-metadata.mdx - page: "Loading data to DevRev" slug: loading-data path: ../docs/pages/airdrop/loading-data.mdx From f7648c71db6c0b3d0318ad04e5ef6f872ff34c99 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Patricija=20Bre=C4=8Dko?= Date: Fri, 28 Mar 2025 11:09:45 +0100 Subject: [PATCH 19/27] Some text fixes --- ...mdx => connect-to-the-external-system.mdx} | 0 .../external-sync-units-extraction.mdx | 19 ++++++++--------- .../airdrop/extracting-data-and-metadata.mdx | 4 +--- fern/docs/pages/airdrop/extracting-data.mdx | 19 ----------------- .../airdrop/extraction-getting-started.mdx | 15 +++++++++++++ .../pages/airdrop/snap-in-configuration.mdx | 21 +++++++++---------- fern/versions/public.yml | 20 +++++++++--------- 7 files changed, 45 insertions(+), 53 deletions(-) rename fern/docs/pages/airdrop/{connecting-to-the-external-system.mdx => connect-to-the-external-system.mdx} (100%) delete mode 100644 fern/docs/pages/airdrop/extracting-data.mdx create mode 100644 fern/docs/pages/airdrop/extraction-getting-started.mdx diff --git a/fern/docs/pages/airdrop/connecting-to-the-external-system.mdx b/fern/docs/pages/airdrop/connect-to-the-external-system.mdx similarity index 100% rename from fern/docs/pages/airdrop/connecting-to-the-external-system.mdx rename to fern/docs/pages/airdrop/connect-to-the-external-system.mdx diff --git a/fern/docs/pages/airdrop/external-sync-units-extraction.mdx b/fern/docs/pages/airdrop/external-sync-units-extraction.mdx index 891f8ec0..7fd5aa36 100644 --- a/fern/docs/pages/airdrop/external-sync-units-extraction.mdx +++ b/fern/docs/pages/airdrop/external-sync-units-extraction.mdx @@ -13,16 +13,7 @@ External sync unit extraction is executed only during the initial import. This phase should be implemented in the [`external-sync-units-extraction.ts`](https://github.com/devrev/adaas-template/blob/main/code/src/functions/extraction/workers/external-sync-units-extraction.ts) file. -The snap-in should emit the list of external sync units in the provided event message -`event_data.external_sync_units` containing the following fields: - -- `id`: The unique identifier in the external system. -- `name`: The human-readable name in the external system. -- `description`: The short description if the external system provides it. -- `item_count`: The number of items (issues, tickets, comments or others) in the external system. - Item count should be provided if it can be obtained in a lightweight manner, such as by calling an API endpoint. - If there is no such way to get it (for example, if the items would need to be extracted to count them), - then the item count should be `-1` to avoid blocking the import with long-running queries. +The snap-in should emit the list of external sync units in the given format: ```typescript const externalSyncUnits: ExternalSyncUnit[] = [ @@ -35,6 +26,14 @@ const externalSyncUnits: ExternalSyncUnit[] = [ ]; ``` +- `id`: The unique identifier in the external system. +- `name`: The human-readable name in the external system. +- `description`: The short description if the external system provides it. +- `item_count`: The number of items (issues, tickets, comments or others) in the external system. + Item count should be provided if it can be obtained in a lightweight manner, such as by calling an API endpoint. + If there is no such way to get it (for example, if the items would need to be extracted to count them), + then the item count should be `-1` to avoid blocking the import with long-running queries. + The snap-in must respond to Airdrop with a message, which contains a list of external sync units as a payload: ```typescript diff --git a/fern/docs/pages/airdrop/extracting-data-and-metadata.mdx b/fern/docs/pages/airdrop/extracting-data-and-metadata.mdx index 0b8c99ca..e03d0917 100644 --- a/fern/docs/pages/airdrop/extracting-data-and-metadata.mdx +++ b/fern/docs/pages/airdrop/extracting-data-and-metadata.mdx @@ -1,5 +1,3 @@ -### Metadata and data extraction - It is recommended that you develop the metadata and data extraction phases iteratively. The idea is that you first try to get one type of object with a @@ -23,7 +21,7 @@ await adapter.emit(ExtractorEventType.ExtractionMetadataDone); Metadata extraction should be implemented in the [metadata-extraction.ts](https://github.com/devrev/adaas-template/blob/main/code/src/functions/extraction/workers/metadata-extraction.ts) file. -#### Preparing `external_domain_metadata.json` +### Preparing `external_domain_metadata.json` This `external_domain_metadata.json` file provides a structured way of describing the external system's domain system, including its domain entities, types, relationships, and other metadata. diff --git a/fern/docs/pages/airdrop/extracting-data.mdx b/fern/docs/pages/airdrop/extracting-data.mdx deleted file mode 100644 index f4232f41..00000000 --- a/fern/docs/pages/airdrop/extracting-data.mdx +++ /dev/null @@ -1,19 +0,0 @@ -Data extraction involves four sequential phases: - -1. external sync units extraction (only on initial import), -2. metadata extraction, -3. data extraction, -4. attachments extraction. - -### Before you start implementation - -It is recommended you consider the following points before you start the implementation: - -1. Examine the available objects in both the external system and DevRev and decide how they should correspond to each other. -2. Identify required fields in DevRev and available fields in the external system API. Plan mappings accordingly, considering how to handle required fields in one system that are optional in the other. -3. Check the external system API too see what are options and limitations. - - - Full specification of objects available in DevRev is - [here](https://github.com/devrev/airdrop-recipe-manager/blob/main/v2/devrev_metadata/devrev_metadata_docs.md) - diff --git a/fern/docs/pages/airdrop/extraction-getting-started.mdx b/fern/docs/pages/airdrop/extraction-getting-started.mdx new file mode 100644 index 00000000..674dd54a --- /dev/null +++ b/fern/docs/pages/airdrop/extraction-getting-started.mdx @@ -0,0 +1,15 @@ +Data extraction involves four sequential phases: + +1. external sync units extraction (only on initial import), +2. metadata extraction, +3. data extraction, +4. attachments extraction. + +### Before you start implementation + +It is recommended you consider the following points before you start the implementation: + +1. Examine the domain model of the external system and identify available and required fields for its objects. +2. Examine DevRev's domain model - use the specification given [here](https://github.com/devrev/airdrop-recipe-manager/blob/main/v2/devrev_metadata/devrev_metadata_docs.md). +3. Decide how objects should correspond to each other and plan field mappings accordingly. Consider how to handle required fields in one system that are optional in the other. +4. Check the external system API too see what are options and limitations. diff --git a/fern/docs/pages/airdrop/snap-in-configuration.mdx b/fern/docs/pages/airdrop/snap-in-configuration.mdx index 57d02640..44ff671a 100644 --- a/fern/docs/pages/airdrop/snap-in-configuration.mdx +++ b/fern/docs/pages/airdrop/snap-in-configuration.mdx @@ -1,7 +1,6 @@ -Development of a snap-in starts with a `manifest.yaml`. -Manifest holds a snap-in configuration: snap-in name, description, version, connection information and information about implemented functionality. +Start developing a snap-in with a `manifest.yaml` file. The manifest defines the snap-in's configuration, including its name, description, version, connection details, and functionality. -Begin by setting the values in the template for `name` and `description`. +Begin by setting the values for `name` and `description` in the template. ```yaml name: ADaaS Template @@ -10,9 +9,9 @@ description: A template for creating a new ADaaS snap-in ### Set snap-in functionalities -The DevRev platform enables the creation of a snap-in that can import data from an external system into DevRev (called _extraction_) and export data from DevRev to an external system (called _loading_). Implementation of extration is necessary, while of loading isn't. +DevRev supports creating snap-ins for importing data from an external system into DevRev (_extraction_) and exporting data from DevRev to an external system (_loading_). Implementing the extraction functionality is essential, while loading is optional. -To specify the implemented functionalities, first review the `functions` section of `manifest.yaml`. +To define the functionalities, review the `functions` section in the manifest. ```yaml functions: @@ -24,9 +23,9 @@ functions: description: Create blueprint and install initial domain mapping ``` -In case you're not going to implement loading, remove it from the list. This will also remove this option from the UI. +If loading is not implemented, remove it from the list. This action also removes the option from the UI. -Next, check the `imports` section. +Check the `imports` section next. ```yaml imports: @@ -39,10 +38,10 @@ imports: - example-connection ``` -Change `slug`, `display_name` and `description` according to your snap-in. +Update the `slug`, `display_name`, and `description` to match your snap-in. -Remove `loader_function` in case you're not using it. +If loading is not used, remove `loader_function`. -Make sure the names for `extractor_function` and `loader_function` match your settings in the `functions` section. +Ensure that `extractor_function` and `loader_function` names correspond with those in the `functions` section. -When you are done you can continue with the next section that focuses on establishing a connection to the external system using `keyring_types`. +Proceed to the next section, which dives in on establishing a connection to the external system using `keyring_types`. diff --git a/fern/versions/public.yml b/fern/versions/public.yml index 1651eca4..7d6c2158 100644 --- a/fern/versions/public.yml +++ b/fern/versions/public.yml @@ -151,21 +151,21 @@ navigation: - page: "Snap-in configuration" slug: snap-in-configuration path: ../docs/pages/airdrop/snap-in-configuration.mdx - - page: "Connecting to the external system" - slug: connecting-to-the-external-system - path: ../docs/pages/airdrop/connecting-to-the-external-system.mdx - - section: Extracting data from external system - slug: extracting-data + - page: "Connect to the external system" + slug: connect-to-the-external-system + path: ../docs/pages/airdrop/connect-to-the-external-system.mdx + - section: Extraction + slug: extraction contents: - - page: "Getting started with extraction" + - page: "Getting started" slug: getting-started - path: ../docs/pages/airdrop/extracting-data.mdx + path: ../docs/pages/airdrop/extraction-getting-started.mdx - page: "External sync units extraction" slug: external-sync-units-extraction path: ../docs/pages/airdrop/external-sync-units-extraction.mdx - - page: "Extracting metadata and data" - slug: extracting-data - path: ../docs/pages/airdrop/extracting-data-and-metadata.mdx + - page: "Data extraction" + slug: data-extraction + path: ../docs/pages/airdrop/data-extraction.mdx - page: "Loading data to DevRev" slug: loading-data path: ../docs/pages/airdrop/loading-data.mdx From 316a82e0f4fe7263c48db59158b857205b8072a4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Patricija=20Bre=C4=8Dko?= Date: Mon, 31 Mar 2025 08:51:31 +0200 Subject: [PATCH 20/27] Add metadata and data extraction --- fern/docs/pages/airdrop/data-extraction.mdx | 79 +++-- .../airdrop/extracting-data-and-metadata.mdx | 41 --- .../airdrop/extraction-getting-started.mdx | 2 +- .../pages/airdrop/metadata-extraction.mdx | 296 +----------------- fern/docs/pages/airdrop/state-handling.mdx | 54 ---- fern/versions/public.yml | 6 + 6 files changed, 72 insertions(+), 406 deletions(-) delete mode 100644 fern/docs/pages/airdrop/extracting-data-and-metadata.mdx delete mode 100644 fern/docs/pages/airdrop/state-handling.mdx diff --git a/fern/docs/pages/airdrop/data-extraction.mdx b/fern/docs/pages/airdrop/data-extraction.mdx index 4e24a88d..3e3974b7 100644 --- a/fern/docs/pages/airdrop/data-extraction.mdx +++ b/fern/docs/pages/airdrop/data-extraction.mdx @@ -1,4 +1,3 @@ - In the data extraction phase, the extractor is expected to call the external system's APIs to retrieve all the items that were updated since the start of the last extraction. If there was no previous extraction (the current run is an initial import), @@ -10,46 +9,45 @@ so that it can extract only items created or updated since this date in the next ## Triggering event Airdrop initiates data extraction by starting the snap-in with a message with event of type -`EXTRACTION_DATA_START` when transitioning to the data extraction phase. +`ExtractionDataStart` when transitioning to the data extraction phase. During the data extraction phase, the snap-in extracts data from an external system, prepares batches of data and uploads them in the form of artifacts to DevRev. -The snap-in must respond to Airdrop with a message with event of type `EXTRACTION_DATA_PROGRESS`, +The snap-in must respond to Airdrop with a message with event of type `ExtractionDataProgress`, together with an optional progress estimate and relevant artifacts when it extracts some data and the maximum Airdrop snap-in runtime (12 minutes) has been reached. If the extraction has been rate-limited by the external system and back-off is required, the snap-in -must respond to Airdrop with a message with event of type `EXTRACTION_DATA_DELAY` and specifying +must respond to Airdrop with a message with event of type `ExtractionDataDelay` and specifying back-off time with `delay` attribute. -In both cases, Airdrop starts the snap-in with a message with event of type `EXTRACTION_DATA_CONTINUE`. -The restarting is immediate (in case of `EXTRACTION_DATA_PROGRESS`) or delayed -(in case of `EXTRACTION_DATA_DELAY`). +In both cases, Airdrop starts the snap-in with a message with event of type `ExtractionDataContinue`. +The restarting is immediate (in case of `ExtractionDataProgress`) or delayed +(in case of `ExtractionDataDelay`). Once the data extraction is done, the snap-in must respond to Airdrop with a message with event of -type `EXTRACTION_DATA_DONE`. +type `ExtractionDataDone`. If data extraction failed in any moment of extraction, the snap-in must respond to Airdrop with a -message with event of type `EXTRACTION_DATA_ERROR`. +message with event of type `ExtractionDataError`. + +Data extraction should be implemented in the [data-extraction.ts](https://github.com/devrev/adaas-template/blob/main/code/src/functions/extraction/workers/data-extraction.ts) file. ## Response from the snap-in -During the data extraction phase, the snap-in uploads batches of extracted items (the recommended -batch size is 2000 items) formatted in JSONL (JSON Lines format), gzipped, and submitted as an -artifact to S3Interact (with tooling from `@devrev/adaas-sdk`). +During the data extraction phase, the snap-in uploads batches of extracted items (with tooling from `@devrev/adaas-sdk`). Each artifact is submitted with an `item_type`, defining a separate domain object from the external system and matching the `record_type` in the provided metadata. -Item types defined when uploading extracted data must validate the declarations in the metadata file. Extracted data must be normalized. - Null values: All fields without a value should either be omitted or set to null. -For example, if an external system provides values such as "", -1 for missing values, -those must be set to null. + For example, if an external system provides values such as "", -1 for missing values, + those must be set to null. - Timestamps: Full-precision timestamps should be formatted as RFC3999 (`1972-03-29T22:04:47+01:00`), -and dates should be just `2020-12-31`. + and dates should be just `2020-12-31`. - References: references must be strings, not numbers or objects. - Number fields must be valid JSON numbers (not strings). - Multiselect fields must be provided as an array (not CSV). @@ -60,23 +58,48 @@ All other fields are contained within the `data` attribute. ```json { - "id": "2102e01F", - "created_date": "1972-03-29T22:04:47+01:00", - "modified_date": "1970-01-01T01:00:04+01:00", - "data": { - "actual_close_date": "1970-01-01T02:33:18+01:00", - "creator": "b8", - "owner": "A3A", - "rca": null, - "severity": "fatal", - "summary": "Lorem ipsum" - } + "id": "2102e01F", + "created_date": "1972-03-29T22:04:47+01:00", + "modified_date": "1970-01-01T01:00:04+01:00", + "data": { + "actual_close_date": "1970-01-01T02:33:18+01:00", + "creator": "b8", + "owner": "A3A", + "rca": null, + "severity": "fatal", + "summary": "Lorem ipsum" + } } ``` - Extracted artifacts can be validated with the `chef-cli` using the following command: ```bash $ chef-cli validate-metadata -m external_domain_metadata.json -r issue < extractor_issues_2.json ``` + +## State handling + +Since each snap-in invocation is a separate runtime instance (with a maximum execution time of 12 minutes), it does not know what has been previously accomplished or how many records have already been extracted. To enable information passing between invocations and runs, support has been added for saving a limited amount of data as the snap-in `state`. Snap-in state persists between phases in one sync run as well as between multiple sync runs. + +A snap-in must consult its state to obtain information on when the last successful forward sync started. + +- The snap-in's `state` is loaded at the start of each invocation and saved at its end. +- The snap-in's `state` must be a valid JSON object. +- Each sync direction (to DevRev and from DevRev) has its own `state` object that is not shared. +- The snap-in `state` should be smaller than 1 MB, which maps to approximately 500,000 characters. + +Effective use of the state and breaking down the problem into smaller chunks are crucial for good performance and user experience. Without knowing what has been processed, the snap-in will extract the same data multiple times, using valuable API capacity and time, and possibly duplicate the data inside DevRev or the external application. + +The snap-in starter template contains an [example](https://github.com/devrev/adaas-template/blob/main/code/src/functions/extraction/index.ts) of a very simple state. Adding more data to the state can help with pagination and rate limiting by saving the point at which extraction was left off. + +To more easily test the state in development, you can decrease the timeout between snap-in invocations. + +```typescript +await spawn({ + ..., + option: { + timeout: 1 * 60 * 1000; // 1 minute in milliseconds + } +}); +``` diff --git a/fern/docs/pages/airdrop/extracting-data-and-metadata.mdx b/fern/docs/pages/airdrop/extracting-data-and-metadata.mdx deleted file mode 100644 index e03d0917..00000000 --- a/fern/docs/pages/airdrop/extracting-data-and-metadata.mdx +++ /dev/null @@ -1,41 +0,0 @@ - - It is recommended that you develop the metadata and data extraction phases - iteratively. The idea is that you first try to get one type of object with a - few simple fields, then continue to extend functionality and test the - implementation throughout the process. - - -The idea for metadata extraction phase is that the snap-in provides an -`external_domain_metadata.json` file to Airdrop. - -```typescript -import externalDomainMetadata from "../../external-system/external_domain_metadata.json"; - -// ... - -await adapter - .getRepo("external_domain_metadata") - ?.push([externalDomainMetadata]); -await adapter.emit(ExtractorEventType.ExtractionMetadataDone); -``` - -Metadata extraction should be implemented in the [metadata-extraction.ts](https://github.com/devrev/adaas-template/blob/main/code/src/functions/extraction/workers/metadata-extraction.ts) file. - -### Preparing `external_domain_metadata.json` - -This `external_domain_metadata.json` file provides a structured way of describing the external system's domain system, -including its domain entities, types, relationships, and other metadata. - -Check the [example](https://github.com/devrev/adaas-template/blob/main/code/src/functions/external-system/external_domain_metadata.json) from the starter template. - -The `record_types` object should contain all the objects that will be imported from the external system and specifications for their fields. - -#### Data extraction - -In initial sync we need to extract all data to create a baseline while in subsequent runs only updated objects need to be extracted. - -Snap-in should consult its internal state to determine what is already extracted and what it did not - -#### Attachments extraction - -For the attachment extraction phase of the import process, the extractor has to upload each attachment to DevRev and associate it with the data object it is attached to. diff --git a/fern/docs/pages/airdrop/extraction-getting-started.mdx b/fern/docs/pages/airdrop/extraction-getting-started.mdx index 674dd54a..790a2e84 100644 --- a/fern/docs/pages/airdrop/extraction-getting-started.mdx +++ b/fern/docs/pages/airdrop/extraction-getting-started.mdx @@ -1,4 +1,4 @@ -Data extraction involves four sequential phases: +Extraction generally involves four sequential phases: 1. external sync units extraction (only on initial import), 2. metadata extraction, diff --git a/fern/docs/pages/airdrop/metadata-extraction.mdx b/fern/docs/pages/airdrop/metadata-extraction.mdx index 9fb32979..912b5dd9 100644 --- a/fern/docs/pages/airdrop/metadata-extraction.mdx +++ b/fern/docs/pages/airdrop/metadata-extraction.mdx @@ -1,295 +1,27 @@ - -During the metadata extraction phase, the Airdrop snap-in must provide an -`external_domain_metadata.json` file on each sync run. +During the metadata extraction phase, the snap-in must provide an +`external_domain_metadata.json` file to Airdrop. This file provides a structured way of describing the external system's domain system, including its domain entities, types, relationships, and other metadata. -The extraction function of the snap-in must provide a valid metadata file. -DevRev provides a JSON schema and a CLI tool [chef-cli](https://github.com/devrev/adaas-chef-cli) -to validate the proposed schema. - ## Triggering event Airdrop initiates the metadata extraction by starting the snap-in with a message with an event -of type `EXTRACTION_METADATA_START`. +of type `ExtractionMetadataStart`. -The snap-in must respond to Airdrop with a message with an event of type `EXTRACTION_METADATA_DONE` -when done, or `EXTRACTION_METADATA_ERROR` in case of an error. +The snap-in must respond to Airdrop with a message with an event of type `ExtractionMetadataDone` +when done, or `ExtractionMetadataError` in case of an error. ## Response from the snap-in -During the metadata extraction phase, the Airdrop snap-in must provide an -`external_domain_metadata.json` file on each sync run. - -The transformation can be crafted and finalized further using the `chef-cli` to ensure extracted -data is mapped consistently to the DevRev domain model. - -## Craft the metadata declaration - -Since crafting metadata declaration in the form of an `external_domain_metadata.json` file can be a -tedious process, a step-by-step approach is useful for understanding the metadata declarations and -as a checklist to declare the metadata for an extraction from a specific external system. - -Metadata declarations include both _static declarations_, formulated by deduction and comparison of -external domain system, and DevRev domain system and _dynamic declarations_ that are obtained during -a snap-in run from external system APIs (since they are configurable in the external system and can -be changed by the end user at any time, such as mandatory fields or custom fields). - - - - ### Declare the extracted record types - - _Record types_ are the types of records that has a well-defined schema you extract from or load - to the external system, a domain object in the external system. - - If the snap-in is extracting issues and comments, a good starting point to declare record types - in `external_domain_metadata.json` would be: - - ```json - { - "record_types": { - "issues": {}, - "comments": {} - } - } - ``` - Although the declaration of record types is arbitrary, they must match the `item_type` - field in the artifacts you will upload. - - ### Declare the custom record types - - If the external system supports custom types, or custom variants of some base record type, - and you want to airdrop those too, you have to declare them in the metadata at runtime. - That is, the extractor will use APIs of the external system to dynamically discover what - custom record types exist. - - The output of this process might look like this: - - ```json - { - "record_types": { - "issues_stock_epic": {}, - "issues_custom2321": {}, - "issues_custom2322": {}, - "comments": {} - } - } - ``` - - ### Provide human-readable names to external record types - - Define human-readable names for the record types defined in your metadata file. - - ```json - { - "record_types":{ - "issues_stock_epic":{ - "name":"Epic" - }, - "issues_custom2321":{ - "name":"Incident report" - }, - "issues_custom2322":{ - "name":"Problem" - }, - "comments":{ - "name":"Comment" - } - } - } - ``` - - ### Categorize external record types - - The metadata allows each external record type to be annotated with one category. - The category key can be an arbitrary string, but it must match the categories declared under - `record_type_categories`. - - Categories of external record types simplify mappings so that a mapping can be applied to a - whole category of record types. - Categories also provide a way how custom record types can be mapped. - - If the external system allows records to change the record type within the category, - while still preserving identity, this can be defined by the `are_record_type_conversions_possible` - field in the `record_type_categories` section. For example, if an issue that can be moved to - become a problem in the external system. - - ```json - { - "record_types":{ - "issues_stock_epic":{ - "name":"Epic", - "category":"issue" - }, - "issues_custom2321":{ - "name":"Incident report", - "category":"issue" - }, - "issues_custom2322":{ - "name":"Problem", - "category":"issue" - }, - "comments":{ - "name":"Comment" - } - }, - "record_type_categories":{ - "issue":{ - "are_record_type_conversions_possible":true - } - } - } - ``` - - ### Declare fields for each record type - - Fields' keys must match what is actually found in the extracted data in the artifacts. - - The supported types are: - - `bool` - - `int` - - `float` - - `text` - - `rich_text`: Formatted text with mentions and images. - - `reference`: IDs referring to another record. References have to declare what they can refer - to, which can be one or more record types (`#record:`) or categories (`#category:`). - - `enum`: A string from a predefined set of values with the optional human-readable names for each value. - - `date` - - `timestamp` - - `struct` - - If the external system supports custom fields, the set of custom fields in each record type - you wish to extract must be declared too. - - Enum fields' set of possible values can often be customizable. - A good practice is to retrieve the set of possible values for all enum fields from the external - system's APIs in each sync run. - - `ID` (primary key) of the record, `created_date`, and `modified_date` must not be declared. - - Example: - - ```json - { - "record_types": { - "issues_stock_epic": { - "name": "Epic", - "category": "issue", - "fields": { - "actual_close_date": { - "name": "Closed at", - "type": "timestamp" - }, - "owner": { - "is_required": true, - "type": "reference", - "reference": { - "refers_to": { - "#record:user": {} - } - } - }, - "creator": { - "is_required": true, - "type": "reference", - "reference": { - "refers_to": { - "#record:user": {} - } - } - }, - "priority": { - "name": "Priority", - "is_required": true, - "type": "enum", - "enum": { - "values": [ - { - "key": "P-0", - "name": "Super important" - }, - { - "key": "P-1" - }, - { - "key": "P-2" - } - ] - } - }, - "target_close_date": { - "type": "date" - }, - "headline": { - "name": "Headline", - "is_required": true, - "type": "text" - } - } - } - } - } - ``` - - ### Declare arrays - - If the field is array in the extracted data, it is still typed with the one of the supported types. - Lists must be marked as a `collection`. - - ``` json - { - "name": "Assignees", - "is_required": true, - "type": "reference", - "reference": { - "refers_to": { - "#category:agents": {} - } - }, - "collection": { - "max_length": 5 - } - } - ``` +Metadata extraction should be implemented in the [metadata-extraction.ts](https://github.com/devrev/adaas-template/blob/main/code/src/functions/extraction/workers/metadata-extraction.ts) file. - ### Handle special references +```typescript +import externalDomainMetadata from "../../external-system/external_domain_metadata.json"; - - Some references have role of parent or child. - This means that the child record doesn't make sense without its parent, for example a comment - attached to a ticket. - Assigning a `reference_type` helps Airdrop correctly handle such fields in case the end-user - decides to filter some of the parent records out. +// ... - - Sometimes the external system uses references besides the primary key of records, for example - when referring to a case by serial number, or to a user by their email. - To correctly resolve such references, they must be marked with 'by_field', which must be a field - existing in that record type, marked 'is_identifier'. For example: - ```json - { - "record_types": { - "users": { - "fields": { - "email": { - "type": "text", - "is_identifier":true - } - } - }, - "comments": { - "fields": { - "user_email": { - "type": "reference", - "reference": { - "refers_to": { - "#record:users": { - "by_field": "email" - } - } - } - } - } - } - } -} - ``` - +await adapter + .getRepo("external_domain_metadata") + ?.push([externalDomainMetadata]); +await adapter.emit(ExtractorEventType.ExtractionMetadataDone); +``` diff --git a/fern/docs/pages/airdrop/state-handling.mdx b/fern/docs/pages/airdrop/state-handling.mdx deleted file mode 100644 index 894d4180..00000000 --- a/fern/docs/pages/airdrop/state-handling.mdx +++ /dev/null @@ -1,54 +0,0 @@ -# Snap-in `state` - -Since each snap-in invocation is a separate runtime instance, it does not know what has been previously accomplished. - -To enable information passing between invocations and runs we have added support for saving a limited amount of data as snap-in `state`. - -* Snap-in's `state` is loaded at the start of each invocation and saved at its end. -* Each sync direction (to DevRev and from DevRev) has its own `state` object that is not shared. -* Snap-in `state` should be smaller than 1 MB.Hard cap is a bit higher. 1 MB maps to approximately 500 000 characters. - -Effective use of the `state` and breaking down the problem into smaller chunks is crucial for good performance and user experience. - -Without knowing what has been processed, the snap-in will extract the same data multiple times, using precious API capacity, time, and posibly duplicate the data inside DevRev or the external application. Duplicate data will lead to frustrating snap-in user experience. - -# Data queries - -First step is to research the external system of records (SOR) public API capabilities. - -* Does it offer searching on created date, modified date, some append only change log? -* How do records relate to each other, can I query them independantly? -* Do records contain parent and/or children information. -* Does a list call return only a summary or all needed information. Do I need to create separate calls to retrieve more than X comments, or threaded comments? -* How many documents does a list call return? Is it best to limit them due to low API limits? - -# Example system - -Lets define the limits of a theoretical system: - -* Allows list queries by the changed date that returns 50 items in one batch. -* Each item contains up to 10 comments, for more you need to use a different API endpoint. -* Retrieved item contains a flag indicating the existence of attachments. - -This simple system has a straightforward data template with a few API limits. - -* Parent item can be retrieved with a list call. -* If the parent contains 10 comments, we have to perform a separate call to see if more comments exist. -* In the separate attachment phase we have to list though items again and extract attachments from those that exist. -* We cannot save all the IDs of parent objects with attachments as that could breach `state` size limits. -* It is best for the `state` to contain last changed date timestamp plus some item IDs in case multiple items were changed at the same time. All 50 item IDs could easily fit in the `state`. -* Between list calls some items can be updated. Two queries with the same changed date might not return the same items. - -# Set the `state` - -In our theoretical system we would be able to batch calls in the increments of 50. - -* In each batch we would extract items and all of their comments. - -After each batch is processed, changed date, item IDs and the changed date of the last seen item is saved in the `state`. -In the next batch we can use the changed date of the last seen item as the starting point. If we are afraid to lose some items, we can always decrement the smallest time chunk the platform supports to make sure some items overlap from one batch to another. -We also have to make sure to not trap ourselves into the cycle to always call with the same changed date and forever retrieve the same items. - -In the same way as we determined to use changed date as a discriminator between chunks we have to make sure we can use an indicator to continue between invocation and between sync runs. - -**As we have to list through all items to extract attachments, we need to make sure that we do not override the changed date and item IDs state between attachment and data phase themselves**. \ No newline at end of file diff --git a/fern/versions/public.yml b/fern/versions/public.yml index 7d6c2158..f2bb08fd 100644 --- a/fern/versions/public.yml +++ b/fern/versions/public.yml @@ -163,9 +163,15 @@ navigation: - page: "External sync units extraction" slug: external-sync-units-extraction path: ../docs/pages/airdrop/external-sync-units-extraction.mdx + - page: "Metadata extraction" + slug: metadata-extraction + path: ../docs/pages/airdrop/metadata-extraction.mdx - page: "Data extraction" slug: data-extraction path: ../docs/pages/airdrop/data-extraction.mdx + - page: "Attachments extraction" + slug: extract-attachments + path: ../docs/pages/airdrop/attachments-extraction.mdx - page: "Loading data to DevRev" slug: loading-data path: ../docs/pages/airdrop/loading-data.mdx From e8d766a2476f5419cb85a4b27bbfc8f6e3eb429a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Patricija=20Bre=C4=8Dko?= Date: Mon, 31 Mar 2025 23:50:35 +0200 Subject: [PATCH 21/27] Shuffle things around --- .../pages/airdrop/attachments-extraction.mdx | 3 + fern/docs/pages/airdrop/data-extraction.mdx | 6 +- fern/docs/pages/airdrop/extraction-phases.mdx | 67 +-------- fern/docs/pages/airdrop/getting-started.mdx | 135 ++++++------------ .../airdrop/how-does-the-snap-in-work.mdx | 50 ------- ...to-the-external-system.mdx => keyring.mdx} | 0 fern/docs/pages/airdrop/local-development.mdx | 67 +++++++++ ...snap-in-configuration.mdx => manifest.mdx} | 2 +- .../pages/airdrop/metadata-extraction.mdx | 2 + fern/docs/pages/airdrop/overview.mdx | 5 +- .../pages/airdrop/publish-to-marketplace.mdx | 2 +- fern/docs/pages/airdrop/snap-in-template.mdx | 40 ++++++ fern/versions/public.yml | 79 ++++------ 13 files changed, 194 insertions(+), 264 deletions(-) delete mode 100644 fern/docs/pages/airdrop/how-does-the-snap-in-work.mdx rename fern/docs/pages/airdrop/{connect-to-the-external-system.mdx => keyring.mdx} (100%) create mode 100644 fern/docs/pages/airdrop/local-development.mdx rename fern/docs/pages/airdrop/{snap-in-configuration.mdx => manifest.mdx} (89%) create mode 100644 fern/docs/pages/airdrop/snap-in-template.mdx diff --git a/fern/docs/pages/airdrop/attachments-extraction.mdx b/fern/docs/pages/airdrop/attachments-extraction.mdx index a24b27bc..1d05c6e4 100644 --- a/fern/docs/pages/airdrop/attachments-extraction.mdx +++ b/fern/docs/pages/airdrop/attachments-extraction.mdx @@ -1,3 +1,5 @@ +For the attachment extraction phase of the import process, the extractor has to upload each attachment to DevRev and associate it with the data object it is attached to. + For the attachment extraction phase of the import process, the extractor has to upload each attachment to DevRev's S3 using the `S3Interact` API. @@ -43,6 +45,7 @@ The uploaded artifact is structured like a normal artifact containing extracted ## Examples Here is an example of an SSOR attachment file: + ```json lines { "id": { diff --git a/fern/docs/pages/airdrop/data-extraction.mdx b/fern/docs/pages/airdrop/data-extraction.mdx index 3e3974b7..4e4c11f0 100644 --- a/fern/docs/pages/airdrop/data-extraction.mdx +++ b/fern/docs/pages/airdrop/data-extraction.mdx @@ -44,7 +44,7 @@ external system and matching the `record_type` in the provided metadata. Extracted data must be normalized. - Null values: All fields without a value should either be omitted or set to null. - For example, if an external system provides values such as "", -1 for missing values, + For example, if an external system provides values such as "", –1 for missing values, those must be set to null. - Timestamps: Full-precision timestamps should be formatted as RFC3999 (`1972-03-29T22:04:47+01:00`), and dates should be just `2020-12-31`. @@ -89,9 +89,9 @@ A snap-in must consult its state to obtain information on when the last successf - Each sync direction (to DevRev and from DevRev) has its own `state` object that is not shared. - The snap-in `state` should be smaller than 1 MB, which maps to approximately 500,000 characters. -Effective use of the state and breaking down the problem into smaller chunks are crucial for good performance and user experience. Without knowing what has been processed, the snap-in will extract the same data multiple times, using valuable API capacity and time, and possibly duplicate the data inside DevRev or the external application. +Effective use of the state and breaking down the problem into smaller chunks are crucial for good performance and user experience. Without knowing what has been processed, the snap-in extracts the same data multiple times, using valuable API capacity and time, and possibly duplicates the data inside DevRev or the external application. -The snap-in starter template contains an [example](https://github.com/devrev/adaas-template/blob/main/code/src/functions/extraction/index.ts) of a very simple state. Adding more data to the state can help with pagination and rate limiting by saving the point at which extraction was left off. +The snap-in starter template contains an [example](https://github.com/devrev/adaas-template/blob/main/code/src/functions/extraction/index.ts) of a simple state. Adding more data to the state can help with pagination and rate limiting by saving the point at which extraction was left off. To more easily test the state in development, you can decrease the timeout between snap-in invocations. diff --git a/fern/docs/pages/airdrop/extraction-phases.mdx b/fern/docs/pages/airdrop/extraction-phases.mdx index ed3c1f84..fd8541af 100644 --- a/fern/docs/pages/airdrop/extraction-phases.mdx +++ b/fern/docs/pages/airdrop/extraction-phases.mdx @@ -7,80 +7,20 @@ sync run. Each sync run is comprised out of phases. Phases follow sequentially, and each can consist of one or more invocations of the snap-in. -```mermaid -sequenceDiagram -%%{init: { - 'theme': 'base', - 'themeVariables': { - 'fontFamily': 'Segoe UI', - 'signalColor': '#7d7f7c', - 'signalTextColor': '#7d7f7c' -}}}%% - actor user as User - participant gw as REST API Gateway - participant ad as Airdrop components - participant ee as External extractor - - note over gw,ad: Extract External Sync Units - user ->> gw: Select import connection - gw ->> ad: Start External Sync Unit extraction - ad ->> ee: Start External Sync Unit extraction - ee -->> gw: List of External Sync Units - gw ->> ad: List of External Sync Units - ad ->> gw: List of External Sync Units - gw -->> user: Show available External Sync Units - - - note over gw,ad: Extract Metadata - ad ->> ee: Start Metadata extraction - ee -->> gw: List of extracted metadata artifacts - gw ->> ad: List of extracted metadata artifacts - - note over gw,ad: Extract data - ad ->> ee: Start data extraction - ee -->> gw: List of extracted data artifacts - gw ->> ad: List of extracted data artifacts - - note over ad: Transform and import data into DevRev - - note over gw,ad: Extract attachments - ad ->> ee: Start attachment extraction - ee -->> gw: List of extracted attachments - gw ->> ad: List of extracted attachments - - ad -->> gw: Finished import - gw -->> user: Show finished import and report - - note over gw,ad: Delete data - user ->> gw: Delete import - gw ->> ad: Delete import - ad ->> ee: Delete data - ee -->> gw: Finished deleting data - gw ->> ad: Finished deleting data - - note over gw,ad: Delete attachments - ad ->> ee: Delete attachments - ee -->> gw: Finished deleting attachments - gw ->> ad: Finished deleting attachments - note over ad: Deletes internal data - ad -->> gw: Finished deleting import - gw -->> user: Import deleted -``` - ## Sync modes ### Initial import An _initial import_ is the first import of data from the external system to DevRev. -It is triggered manually by the end user in DevRev's **Imports** UI. +It is triggered manually by the end user in DevRev's **Airdrops** UI. An _initial import_ consists of the following phases: + 1. External sync units extraction 2. Metadata extraction 3. Data extraction 4. Attachments extraction - ### 1-way sync A _1-way sync_ refers to any extraction after the initial import has been successfully completed. @@ -93,15 +33,16 @@ Airdrop snap-ins must maintain their own states that persists between phases in as well as between sync runs. A 1-way sync consists of the following phases: + 1. Metadata extraction 2. Data extraction 3. Attachments extraction A 1-way sync extracts only the domain objects updated or created since the previous successful sync run. - ### Deletion mode A _deletion_ consists of the following phases: + 1. Data deletion 2. Attachments deletion diff --git a/fern/docs/pages/airdrop/getting-started.mdx b/fern/docs/pages/airdrop/getting-started.mdx index e22668a9..b1d9cbd9 100644 --- a/fern/docs/pages/airdrop/getting-started.mdx +++ b/fern/docs/pages/airdrop/getting-started.mdx @@ -1,8 +1,6 @@ -## Before you begin +## DevRev organization -### Create a new organization - -If this is your first time setting up an organization within DevRev, you need to create a dedicated DevRev organization for developing your new Airdrop snap-in: +If this is your first time setting up an organization within DevRev, you need to create a DevRev organization for developing your new Airdrop snap-in: 1. Click on your profile picture in the top left corner. 2. Go to the **Orgs** section and click on **+**. @@ -10,90 +8,45 @@ If this is your first time setting up an organization within DevRev, you need to You now find your newly created organization under **Orgs**. -## Local development - -For ease of development, you can run your snap-in locally and receive logs directly in your console. - -### Prerequisites - -- [devrev-cli](https://developer.devrev.ai/snapin-development/references/cli-install) (version 4.7 or higher) -- [jq](https://stedolan.github.io/jq) -- [golang](https://go.dev/doc/install) -- [nodejs](https://nodejs.org/en/download/package-manager) (version 18.x.x+ or higher) -- [ngrok](https://ngrok.com) - -### Starter template - -DevRev provides a template with example code to implement an Airdrop as a Service (ADaaS) snap-in. - -1. Create a new repository: - - Create a new repository from this template by clicking the "Use this template" button in the upper right corner and then "Create a new repository". - - The repository name must start with `airdrop-` (for example, `airdrop--snap-in`). -2. Open the project in your IDE and authenticate to your DevRev organization using the DevRev CLI: - ```bash - devrev profiles authenticate --org - ``` -3. Install the snap-in dependencies from the `code` directory: - ```bash - npm ci - ``` -4. Start the snap-in development server from the `code` directory: - ```bash - npm run test:server -- local - ``` - This will start the snap-in server on `http://localhost:8000`. -5. Start the ngrok tunnel in a separate terminal window: - ```bash - ngrok http 8000 - ``` - This will create a tunnel to your local server. - The ngrok forwarding URL will be displayed in the terminal window. -6. Copy the ngrok forwarding URL and create a new snap-in version and snap-in package using the DevRev CLI: - ```bash - devrev snap_in_version create-one --manifest ./manifest.yaml --create-package --testing-url - ``` -7. Create a snap-in draft using the DevRev CLI: - ```bash - devrev snap_in draft - ``` -8. After the snap-in draft is created, install the snap-in: - ```bash - devrev snap_in activate - ``` - -### Snap-in code structure - -If you check the template, you can see that a snap-in is a Node.js project with a specific file structure: - -``` -- manifest.yaml -- code/ - - scripts/ - - src/ - - fixtures/ - - functions/ - - external-system/ - - extraction/ - - workers/ - - attachments-extraction.ts - - data-extraction.ts - - external-sync-units-extraction.ts - - metadata-extraction.ts - - index.ts - - install_initial_domain_mapping/ - - loading/ - - workers/ - - load-data.ts - - data-extraction.ts - - external-sync-units-extraction.ts - - metadata-extraction.ts - - index.ts - - test-runner/ - - test/ -``` - -The `manifest.yaml` file plays a critical role in a snap-in, holding key configuration details such as the snap-in's name, description, version, connection information, and functionality specifics. This file is the initial step in the development of a snap-in. - -The `code/` folder holds the implementation of the snap-in. Inside, the `functions/` subfolder includes the logic for retrieving external sync units, extracting data from and loading data to external systems, and mapping between them. - -The `scripts/` folder contains the scripts necessary for the building and deployment of the snap-in, streamlining the development and deployment processes with the help of tools like `deploy.sh` and `build.sh`. +## External system + +Before starting the development of a snap-in for integration with an external system API, consider gathering the following information: + +- API Documentation: Obtain the official API documentation of the external system. This will be the primary source of information about how to connect and interact with the system. +- Authentication and Authorization: Understand the authentication and authorization methods required. This may include API keys, OAuth tokens, or other security mechanisms. +- Endpoints and Resources: Identify the required API endpoints and resources. Understand their functions, input parameters, expected outputs, and usage limitations. +- Data Format: Determine the data format used by the API, such as JSON or XML. This helps in parsing responses and formatting requests appropriately. +- Rate Limits and Quotas: Be aware of any rate limits or usage quotas. This information is crucial to ensure that the integration does not exceed allowed requests or data usage. +- Error Handling: Learn about error response formats and codes. Knowing this will help in effectively handling errors and exceptions in your integration. + +## Terminology + +### Sync run + +Airdrop extractions are done in _sync runs_. +A sync run is one execution of an airdrop or sync. + +### Forward sync + +A _forward sync_ is a sync run from an external system to DevRev. +An **extractor** function in the snap-in is responsible for extracting data from the external system. + +### Reverse sync + +A _reverse sync_ is a sync run from DevRev to an external system. +It uses a **loader** function, to create or update data in the external system. + +### Initial sync + +It is the first forward sync to run. In initial sync all data needs to be extracted to create a baseline (while in incremental runs only updated objects need to be extracted). + +### 1-way (incremental) sync + +A _1-way sync_ (or _incremental sync_) refers to any extraction after the initial sync run has been successfully completed. +In the incremental sync an extractor extracts data that was created or updated in the external system after the start +of the latest successful forward sync, including any changes that occurred during the forward sync, +but were not picked up by it. + +### Sync unit + +A _sync unit_ is one connection instance to an external system (for example SalesForce, Freshdesk, Zendesk). diff --git a/fern/docs/pages/airdrop/how-does-the-snap-in-work.mdx b/fern/docs/pages/airdrop/how-does-the-snap-in-work.mdx deleted file mode 100644 index 165ca357..00000000 --- a/fern/docs/pages/airdrop/how-does-the-snap-in-work.mdx +++ /dev/null @@ -1,50 +0,0 @@ -Now that you have a running snap-in, you can start an airdrop. -Go to UI and click **Airdrops** -> **Start Airdrop** -> **Your snap-in**. - -### What happens in the UI - -1. Starting an airdrop begins with establishing a connection to the external system. The snap-in prompts for specific information needed to create this connection. Afterwards, the snap-in verifies that the connection is successful and you should see a success message. - -2. Once the connection is established, the snap-in collects information from the external system about the data unit to be exported. This phase is called _external sync units extraction_. An external sync unit is the data unit from the external system to be imported to DevRev. At this point, the UI prompts the user to select an external sync unit from the list. - -3. The snap-in then extracts data from the external system. - -4. Afterwards, the UI requests the mapping of data from the external system to DevRev. The user can decide which fields from each system should match. - -5. Once mapping is complete, the snap-in saves the data to DevRev. - -6. Finally, the snap-in streams any possible attachments from the external system to DevRev. - -The process described above is called the **initial sync**. In all subsequent syncs, the first two steps are skipped. - -### What happens in the background - -The snap-in constantly communicates with Airdrop, which manages its behavior. When an action occurs in the UI, Airdrop sends a message to the snap-in with details about the event. The snap-in performs an action based on the event type. For example, when a user starts a new airdrop for the first time, Airdrop instructs the snap-in to perform an external sync unit extraction. The snap-in carries out the task and sends back available sync units to Airdrop. - -## Terminology - -### Sync run - -Airdrop extractions are done in _sync runs_. -A sync run is one execution of an airdrop or sync. - -### Forward sync - -A _forward sync_ is a sync run from an external system to DevRev. -An **extractor** function in the snap-in is responsible for extracting data from the external system. - -### Reverse sync - -A _reverse sync_ is a sync run from DevRev to an external system. -It uses a **loader** function, to create or update data in the external system. - -### Initial sync - -It is the first forward sync to run. In initial sync we need to extract all data to create a baseline (while in incremental runs only updated objects need to be extracted). - -### 1-way (incremental) sync - -A _1-way sync_ (or _incremental sync_) refers to any extraction after the initial sync run has been successfully completed. -In the incremental sync an extractor extracts data that was created or updated in the external system after the start -of the latest successful forward sync, including any changes that occurred during the forward sync, -but were not picked up by it. diff --git a/fern/docs/pages/airdrop/connect-to-the-external-system.mdx b/fern/docs/pages/airdrop/keyring.mdx similarity index 100% rename from fern/docs/pages/airdrop/connect-to-the-external-system.mdx rename to fern/docs/pages/airdrop/keyring.mdx diff --git a/fern/docs/pages/airdrop/local-development.mdx b/fern/docs/pages/airdrop/local-development.mdx new file mode 100644 index 00000000..132a8232 --- /dev/null +++ b/fern/docs/pages/airdrop/local-development.mdx @@ -0,0 +1,67 @@ +For ease of development, you can run your snap-in locally and receive logs directly in your console. + +## Prerequisites + +- [devrev-cli](https://developer.devrev.ai/snapin-development/references/cli-install) (version 4.7 or higher) +- [jq](https://stedolan.github.io/jq) +- [golang](https://go.dev/doc/install) +- [nodejs](https://nodejs.org/en/download/package-manager) (version 18.x.x+ or higher) +- [ngrok](https://ngrok.com) + +## Run the template + +DevRev provides a template with example code to implement an Airdrop snap-in. + +1. Create a new repository: + - Create a new repository from this template by clicking the "Use this template" button in the upper right corner and then "Create a new repository". + - The repository name must start with `airdrop-` (for example, `airdrop--snap-in`). +2. Open the project in your IDE and authenticate to your DevRev organization using the DevRev CLI: + ```bash + devrev profiles authenticate --org + ``` +3. Install the snap-in dependencies from the `code` directory: + ```bash + npm ci + ``` +4. Start the snap-in development server from the `code` directory: + ```bash + npm run test:server -- local + ``` + This will start the snap-in server on `http://localhost:8000`. +5. Start the ngrok tunnel in a separate terminal window: + ```bash + ngrok http 8000 + ``` + This will create a tunnel to your local server. + The ngrok forwarding URL will be displayed in the terminal window. +6. Copy the ngrok forwarding URL and create a new snap-in version and snap-in package using the DevRev CLI: + ```bash + devrev snap_in_version create-one --manifest ./manifest.yaml --create-package --testing-url + ``` +7. Create a snap-in draft using the DevRev CLI: + ```bash + devrev snap_in draft + ``` +8. After the snap-in draft is created, install the snap-in: + ```bash + devrev snap_in activate + ``` + +Now that you have a running snap-in, you can start an airdrop. +Go to DevRev app and click **Airdrops** -> **Start Airdrop** -> **Your snap-in**. + +1. Starting an airdrop begins with establishing a connection to the external system. The snap-in prompts for specific information needed to create this connection. Afterwards, the snap-in verifies that the connection is successful and you should see a success message. + +2. Once the connection is established, the snap-in collects information from the external system about the data unit to be exported. This phase is called _external sync units extraction_. An external sync unit is the data unit from the external system to be imported to DevRev. At this point, the DevRev app prompts the user to select an external sync unit from the list. + +3. The snap-in then extracts data from the external system. + +4. Afterwards, the DevRev app requests the mapping of data from the external system to DevRev. The user can decide which fields from each system should match. + +5. Once mapping is complete, the snap-in saves the data to DevRev. + +6. Finally, the snap-in streams any possible attachments from the external system to DevRev. + +The process described above is called the **initial sync**. In all subsequent syncs, the first two steps are skipped. + +The snap-in constantly communicates with Airdrop, which manages its behavior. When an action occurs in the UI, Airdrop sends a message to the snap-in with details about the event. The snap-in performs an action based on the event type. For example, when a user starts a new airdrop for the first time, Airdrop instructs the snap-in to perform an external sync unit extraction. The snap-in carries out the task and sends back available sync units to Airdrop. diff --git a/fern/docs/pages/airdrop/snap-in-configuration.mdx b/fern/docs/pages/airdrop/manifest.mdx similarity index 89% rename from fern/docs/pages/airdrop/snap-in-configuration.mdx rename to fern/docs/pages/airdrop/manifest.mdx index 44ff671a..a0af8ef2 100644 --- a/fern/docs/pages/airdrop/snap-in-configuration.mdx +++ b/fern/docs/pages/airdrop/manifest.mdx @@ -1,4 +1,4 @@ -Start developing a snap-in with a `manifest.yaml` file. The manifest defines the snap-in's configuration, including its name, description, version, connection details, and functionality. +Start developing your snap-in with a `manifest.yaml` file. The manifest defines the snap-in's configuration, including its name, description, version, connection details, and functionality. Begin by setting the values for `name` and `description` in the template. diff --git a/fern/docs/pages/airdrop/metadata-extraction.mdx b/fern/docs/pages/airdrop/metadata-extraction.mdx index 912b5dd9..77f97f4e 100644 --- a/fern/docs/pages/airdrop/metadata-extraction.mdx +++ b/fern/docs/pages/airdrop/metadata-extraction.mdx @@ -3,6 +3,8 @@ During the metadata extraction phase, the snap-in must provide an This file provides a structured way of describing the external system's domain system, including its domain entities, types, relationships, and other metadata. +Before you start implementation, examine the domain model of the external system to identify available and required fields for its objects. Then, review DevRev's domain model using the specification provided [here](https://github.com/devrev/airdrop-recipe-manager/blob/main/v2/devrev_metadata/devrev_metadata_docs.md). With this information, decide how objects should correspond to each other and plan field mappings accordingly. It's important to consider how to manage required fields in one system that may be optional in the other. + ## Triggering event Airdrop initiates the metadata extraction by starting the snap-in with a message with an event diff --git a/fern/docs/pages/airdrop/overview.mdx b/fern/docs/pages/airdrop/overview.mdx index 895377ef..71594983 100644 --- a/fern/docs/pages/airdrop/overview.mdx +++ b/fern/docs/pages/airdrop/overview.mdx @@ -2,9 +2,6 @@ _Airdrop_ is DevRev’s solution to migrate data. It allows our customers to bri data from external systems to DevRev, export data back to external systems, and keep data in sync between DevRev and the external systems. -DevRev is extending Airdrop's capabilities to give snap-in developers the ability to integrate with -DevRev’s Airdrop functionality. -It enables developers to create external workers (extractors and loaders) to bring data from various -external systems. +DevRev provides a range of snap-ins available through the DevRev marketplace for integration with various external systems like Jira and Zendesk. Sometimes, a specific snap-in may not be available for certain external systems. This is where Airdrop's enhanced functionality comes in, allowing developers to integrate with DevRev’s Airdrop. It enables the creation of external workers (extractors and loaders) to facilitate data transfer from different external systems. You can learn more about Airdrop in the [general documentation](https://docs.devrev.ai/import#airdrop-features). diff --git a/fern/docs/pages/airdrop/publish-to-marketplace.mdx b/fern/docs/pages/airdrop/publish-to-marketplace.mdx index 2844aae6..d836e0df 100644 --- a/fern/docs/pages/airdrop/publish-to-marketplace.mdx +++ b/fern/docs/pages/airdrop/publish-to-marketplace.mdx @@ -10,7 +10,7 @@ devrev marketplace_submissions create ### 2. Submit marketplace listing for review -The newly created marketplace submission is in the Draft state. In order to publish it, we first have to transition it to Waiting for review state. This can be done by running the following command, which starts a wizard to guide you through the process: +The newly created marketplace submission is in the Draft state. In order to publish it, it first has to transition to Waiting for review state. This can be done by running the following command, which starts a wizard to guide you through the process: ```bash devrev marketplace_submissions transition diff --git a/fern/docs/pages/airdrop/snap-in-template.mdx b/fern/docs/pages/airdrop/snap-in-template.mdx new file mode 100644 index 00000000..a8ef4f43 --- /dev/null +++ b/fern/docs/pages/airdrop/snap-in-template.mdx @@ -0,0 +1,40 @@ +For fast and efficient snap-in development, a template is available at [this GitHub repository](https://github.com/devrev/adaas-template). This template is highly recommended for developers as it streamlines the development process. + +The template makes use of DevRev's SDK library to manage communication with Airdrop, simplifying the interaction with DevRev's API. It also ensures that the correct communication protocol with Airdrop is followed. + +Communication between Airdrop and the snap-in involves messages known as events. Depending on the type of event, the snap-in performs different actions. + +### Snap-in code structure + +If you check the template, you can see that a snap-in is a Node.js project with a specific file structure: + +``` +- manifest.yaml +- code/ + - scripts/ + - src/ + - fixtures/ + - functions/ + - external-system/ + - extraction/ + - workers/ + - attachments-extraction.ts + - data-extraction.ts + - external-sync-units-extraction.ts + - metadata-extraction.ts + - index.ts + - install_initial_domain_mapping/ + - loading/ + - workers/ + - load-attachments.ts + - load-data.ts + - index.ts + - test-runner/ + - test/ +``` + +The `manifest.yaml` file plays a critical role in a snap-in, holding key configuration details such as the snap-in's name, description, version, connection information, and functionality specifics. This file is the initial step in the development of a snap-in. + +The `code/` folder holds the implementation of the snap-in. Inside, the `functions/` subfolder includes the logic for retrieving external sync units, extracting data from and loading data to external systems, and mapping between them. + +The `scripts/` folder contains the scripts necessary for the building and deployment of the snap-in, streamlining the development and deployment processes with the help of tools like `deploy.sh` and `build.sh`. diff --git a/fern/versions/public.yml b/fern/versions/public.yml index f2bb08fd..18c9d629 100644 --- a/fern/versions/public.yml +++ b/fern/versions/public.yml @@ -145,72 +145,49 @@ navigation: - page: "Getting started" slug: getting-started path: ../docs/pages/airdrop/getting-started.mdx - - page: "Start an airdrop" - slug: how-does-the-snap-in-work - path: ../docs/pages/airdrop/how-does-the-snap-in-work.mdx - - page: "Snap-in configuration" - slug: snap-in-configuration - path: ../docs/pages/airdrop/snap-in-configuration.mdx - - page: "Connect to the external system" - slug: connect-to-the-external-system - path: ../docs/pages/airdrop/connect-to-the-external-system.mdx - - section: Extraction - slug: extraction - contents: - - page: "Getting started" - slug: getting-started - path: ../docs/pages/airdrop/extraction-getting-started.mdx - - page: "External sync units extraction" - slug: external-sync-units-extraction - path: ../docs/pages/airdrop/external-sync-units-extraction.mdx - - page: "Metadata extraction" - slug: metadata-extraction - path: ../docs/pages/airdrop/metadata-extraction.mdx - - page: "Data extraction" - slug: data-extraction - path: ../docs/pages/airdrop/data-extraction.mdx - - page: "Attachments extraction" - slug: extract-attachments - path: ../docs/pages/airdrop/attachments-extraction.mdx - - page: "Loading data to DevRev" - slug: loading-data - path: ../docs/pages/airdrop/loading-data.mdx - - page: "Examples of snap-ins" - slug: snap-in-examples - path: ../docs/pages/airdrop/snap-in-examples.mdx - - page: "Deploy to organization" - slug: deploy-to-organization - path: ../docs/pages/airdrop/deploy-to-organization.mdx - - page: "Publish to the marketplace" - slug: publish-to-marketplace - path: ../docs/pages/airdrop/publish-to-marketplace.mdx - - page: "Common issues - FAQ" - slug: faq - path: ../docs/pages/airdrop/faq.mdx + - page: "Airdrop Snap-in template" + slug: snap-in-template + path: ../docs/pages/airdrop/snap-in-template.mdx + - page: "Local development" + slug: local-development + path: ../docs/pages/airdrop/local-development.mdx + - page: "Manifest" + slug: manifest + path: ../docs/pages/airdrop/manifest.mdx + - page: "Keyring" + slug: keyring + path: ../docs/pages/airdrop/keyring.mdx - page: "Extraction phases" slug: extraction-phases - hidden: true path: ../docs/pages/airdrop/extraction-phases.mdx - - page: "Loading phases" - slug: loading-phases - hidden: true - path: ../docs/pages/airdrop/loading-phases.mdx - page: "External sync units extraction" slug: external-sync-units-extraction - hidden: true path: ../docs/pages/airdrop/external-sync-units-extraction.mdx - page: "Metadata extraction" - hidden: true slug: metadata-extraction path: ../docs/pages/airdrop/metadata-extraction.mdx - page: "Data extraction" - hidden: true slug: data-extraction path: ../docs/pages/airdrop/data-extraction.mdx - page: "Attachments extraction" - hidden: true slug: extract-attachments path: ../docs/pages/airdrop/attachments-extraction.mdx + - page: "Loading phases" + slug: loading-phases + hidden: true + path: ../docs/pages/airdrop/loading-phases.mdx + - page: "Examples of snap-ins" + slug: snap-in-examples + path: ../docs/pages/airdrop/snap-in-examples.mdx + - page: "Deploy to organization" + slug: deploy-to-organization + path: ../docs/pages/airdrop/deploy-to-organization.mdx + - page: "Publish to the marketplace" + slug: publish-to-marketplace + path: ../docs/pages/airdrop/publish-to-marketplace.mdx + - page: "Common issues - FAQ" + slug: faq + path: ../docs/pages/airdrop/faq.mdx - page: "Data and attachments deletion" slug: data-attachments-deletion hidden: true From ef2f14ca00a4137c6d0c6636e37814130f09dc96 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Patricija=20Bre=C4=8Dko?= Date: Wed, 2 Apr 2025 14:46:31 +0200 Subject: [PATCH 22/27] Add information from SDK readme --- .../pages/airdrop/attachments-extraction.mdx | 3 - fern/docs/pages/airdrop/data-extraction.mdx | 23 +- .../pages/airdrop/deploy-to-organization.mdx | 4 +- .../external-sync-units-extraction.mdx | 6 +- fern/docs/pages/airdrop/extraction-phases.mdx | 94 +++--- fern/docs/pages/airdrop/getting-started.mdx | 61 +++- fern/docs/pages/airdrop/local-development.mdx | 4 +- fern/docs/pages/airdrop/manifest.mdx | 2 +- .../pages/airdrop/metadata-extraction.mdx | 298 ++++++++++-------- fern/docs/pages/airdrop/overview.mdx | 5 +- fern/docs/pages/airdrop/snap-in-template.mdx | 17 +- fern/versions/public.yml | 2 +- 12 files changed, 299 insertions(+), 220 deletions(-) diff --git a/fern/docs/pages/airdrop/attachments-extraction.mdx b/fern/docs/pages/airdrop/attachments-extraction.mdx index 1d05c6e4..85519ee1 100644 --- a/fern/docs/pages/airdrop/attachments-extraction.mdx +++ b/fern/docs/pages/airdrop/attachments-extraction.mdx @@ -1,8 +1,5 @@ For the attachment extraction phase of the import process, the extractor has to upload each attachment to DevRev and associate it with the data object it is attached to. -For the attachment extraction phase of the import process, the extractor has to upload each -attachment to DevRev's S3 using the `S3Interact` API. - ## Triggering event Airdrop initiates the attachment extraction by starting the snap-in with a message with an event of diff --git a/fern/docs/pages/airdrop/data-extraction.mdx b/fern/docs/pages/airdrop/data-extraction.mdx index 20fa89af..1b300463 100644 --- a/fern/docs/pages/airdrop/data-extraction.mdx +++ b/fern/docs/pages/airdrop/data-extraction.mdx @@ -9,28 +9,28 @@ so that it can extract only items created or updated since this date in the next ## Triggering event Airdrop initiates data extraction by starting the snap-in with a message with event of type -`ExtractionDataStart` when transitioning to the data extraction phase. +`EXTRACTION_DATA_START` when transitioning to the data extraction phase. During the data extraction phase, the snap-in extracts data from an external system, prepares batches of data and uploads them in the form of artifacts to DevRev. -The snap-in must respond to Airdrop with a message with event of type `ExtractionDataProgress`, +The snap-in must respond to Airdrop with a message with event of type `EXTRACTION_DATA_PROGRESS`, together with an optional progress estimate and relevant artifacts when it extracts some data and the maximum Airdrop snap-in runtime (12 minutes) has been reached. If the extraction has been rate-limited by the external system and back-off is required, the snap-in -must respond to Airdrop with a message with event of type `ExtractionDataDelay` and specifying +must respond to Airdrop with a message with event of type `EXTRACTION_DATA_DELAY` and specifying back-off time with `delay` attribute. -In both cases, Airdrop starts the snap-in with a message with event of type `ExtractionDataContinue`. -The restarting is immediate (in case of `ExtractionDataProgress`) or delayed -(in case of `ExtractionDataDelay`). +In both cases, Airdrop starts the snap-in with a message with event of type `EXTRACTION_DATA_CONTINUE`. +The restarting is immediate (in case of `EXTRACTION_DATA_PROGRESS`) or delayed +(in case of `EXTRACTION_DATA_DELAY`). Once the data extraction is done, the snap-in must respond to Airdrop with a message with event of -type `ExtractionDataDone`. +type `EXTRACTION_DATA_DONE`. If data extraction failed in any moment of extraction, the snap-in must respond to Airdrop with a -message with event of type `ExtractionDataError`. +message with event of type `EXTRACTION_DATA_ERROR`. Data extraction should be implemented in the [data-extraction.ts](https://github.com/devrev/adaas-template/blob/main/code/src/functions/extraction/workers/data-extraction.ts) file. @@ -42,11 +42,12 @@ Each artifact is submitted with an `item_type`, defining a separate domain objec external system and matching the `record_type` in the provided metadata. Extracted data must be normalized: + - Null values: All fields without a value should either be omitted or set to null. -For example, if an external system provides values such as "", –1 for missing values, -those must be set to null. + For example, if an external system provides values such as "", –1 for missing values, + those must be set to null. - Timestamps: Full-precision timestamps should be formatted as RFC3339 (`1972-03-29T22:04:47+01:00`), -and dates should be just `2020-12-31`. + and dates should be just `2020-12-31`. - References: references must be strings, not numbers or objects. - Number fields must be valid JSON numbers (not strings). - Multiselect fields must be provided as an array (not CSV). diff --git a/fern/docs/pages/airdrop/deploy-to-organization.mdx b/fern/docs/pages/airdrop/deploy-to-organization.mdx index cdbfe3e2..27f54c52 100644 --- a/fern/docs/pages/airdrop/deploy-to-organization.mdx +++ b/fern/docs/pages/airdrop/deploy-to-organization.mdx @@ -14,13 +14,13 @@ This step is also a prerequisite for publishing the snap-in on the DevRev market To observe logs from your snap-in in your development environment: -``` +```bash devrev snap_in_package logs | jq ``` To open logs in your favorite editor: -``` +```bash devrev snap_in_package logs | code - ``` diff --git a/fern/docs/pages/airdrop/external-sync-units-extraction.mdx b/fern/docs/pages/airdrop/external-sync-units-extraction.mdx index 7fd5aa36..34a2e365 100644 --- a/fern/docs/pages/airdrop/external-sync-units-extraction.mdx +++ b/fern/docs/pages/airdrop/external-sync-units-extraction.mdx @@ -52,8 +52,4 @@ await adapter.emit(ExtractorEventType.ExtractionExternalSyncUnitsError, { }); ``` -### UI - -Start a new airdrop in the DevRev App. - -If external sync units extraction is successful, you should be prompted to choose an external sync unit from the list. +To test your changes, start a new airdrop in the DevRev App. If external sync units extraction is successful, you should be prompted to choose an external sync unit from the list. diff --git a/fern/docs/pages/airdrop/extraction-phases.mdx b/fern/docs/pages/airdrop/extraction-phases.mdx index fd8541af..51ef5f6b 100644 --- a/fern/docs/pages/airdrop/extraction-phases.mdx +++ b/fern/docs/pages/airdrop/extraction-phases.mdx @@ -1,48 +1,66 @@ -Airdrop extractions are done in _sync runs_. -A sync run is one execution of an import or sync. -If you do an initial import from the external system to DevRev, that import will be one sync run. -If you then decide to do a reverse sync from DevRev to the external system, that would be another -sync run. +Each snap-in must handle all the phases of Airdrop extraction. In a snap-in,you typically define a run +function that iterates over events and invokes workers per extraction phase. -Each sync run is comprised out of phases. -Phases follow sequentially, and each can consist of one or more invocations of the snap-in. +The SDK library exports `processTask` to structure the work within each phase, and `onTimeout` function +to handle timeouts. -## Sync modes +The snap-in extraction lifecycle consists of three main phases: External Sync Units Extraction, +Metadata Extraction, and Data Extraction. Each phase is defined in a separate file and is responsible +for fetching the respective data. -### Initial import +The SDK library provides a repository management system to handle artifacts in batches. +The `initializeRepos` function initializes the repositories, and the `push` function uploads the +artifacts to the repositories. The `postState` function is used to post the state of the extraction task. -An _initial import_ is the first import of data from the external system to DevRev. -It is triggered manually by the end user in DevRev's **Airdrops** UI. +State management is crucial for snap-ins to maintain the state of the extraction task. +The `postState` function is used to post the state of the extraction task. +The state is stored in the adapter and can be retrieved using the `adapter.state` property. -An _initial import_ consists of the following phases: +```typescript +import { AirdropEvent, EventType, spawn } from "@devrev/ts-adaas"; -1. External sync units extraction -2. Metadata extraction -3. Data extraction -4. Attachments extraction +interface DummyExtractorState { + issues: { completed: boolean }; + users: { completed: boolean }; + attachments: { completed: boolean }; +} -### 1-way sync +const initialState: DummyExtractorState = { + issues: { completed: false }, + users: { completed: false }, + attachments: { completed: false }, +}; -A _1-way sync_ refers to any extraction after the initial import has been successfully completed. -An extractor extracts data that was created or updated in the external system after the start -of the latest successful forward sync, including any changes that occurred during the forward sync, -but were not picked up by it. +function getWorkerPerExtractionPhase(event: AirdropEvent) { + let path; + switch (event.payload.event_type) { + case EventType.ExtractionExternalSyncUnitsStart: + path = __dirname + "/workers/external-sync-units-extraction"; + break; + case EventType.ExtractionMetadataStart: + path = __dirname + "/workers/metadata-extraction"; + break; + case EventType.ExtractionDataStart: + case EventType.ExtractionDataContinue: + path = __dirname + "/workers/data-extraction"; + break; + } + return path; +} -A snap-in must consult its state to get information on when the last successful forward sync started. -Airdrop snap-ins must maintain their own states that persists between phases in a sync run, -as well as between sync runs. +const run = async (events: AirdropEvent[]) => { + for (const event of events) { + const file = getWorkerPerExtractionPhase(event); + await spawn({ + event, + initialState, + workerPath: file, + options: { + isLocalDevelopment: true, + }, + }); + } +}; -A 1-way sync consists of the following phases: - -1. Metadata extraction -2. Data extraction -3. Attachments extraction - -A 1-way sync extracts only the domain objects updated or created since the previous successful sync run. - -### Deletion mode - -A _deletion_ consists of the following phases: - -1. Data deletion -2. Attachments deletion +export default run; +``` diff --git a/fern/docs/pages/airdrop/getting-started.mdx b/fern/docs/pages/airdrop/getting-started.mdx index b1d9cbd9..e60b140d 100644 --- a/fern/docs/pages/airdrop/getting-started.mdx +++ b/fern/docs/pages/airdrop/getting-started.mdx @@ -1,6 +1,7 @@ ## DevRev organization -If this is your first time setting up an organization within DevRev, you need to create a DevRev organization for developing your new Airdrop snap-in: +If this is your first time setting up an organization within DevRev, you need to create a +DevRev organization for developing your new Airdrop snap-in: 1. Click on your profile picture in the top left corner. 2. Go to the **Orgs** section and click on **+**. @@ -10,14 +11,21 @@ You now find your newly created organization under **Orgs**. ## External system -Before starting the development of a snap-in for integration with an external system API, consider gathering the following information: - -- API Documentation: Obtain the official API documentation of the external system. This will be the primary source of information about how to connect and interact with the system. -- Authentication and Authorization: Understand the authentication and authorization methods required. This may include API keys, OAuth tokens, or other security mechanisms. -- Endpoints and Resources: Identify the required API endpoints and resources. Understand their functions, input parameters, expected outputs, and usage limitations. -- Data Format: Determine the data format used by the API, such as JSON or XML. This helps in parsing responses and formatting requests appropriately. -- Rate Limits and Quotas: Be aware of any rate limits or usage quotas. This information is crucial to ensure that the integration does not exceed allowed requests or data usage. -- Error Handling: Learn about error response formats and codes. Knowing this will help in effectively handling errors and exceptions in your integration. +Before starting the development of a snap-in for integration with an external system API, +consider gathering the following information: + +- **API documentation**: Obtain the official API documentation of the external system. + This will be the primary source of information about how to connect and interact with the system. +- **Authentication and authorization**: Understand the authentication and authorization + methods required. This may include API keys, OAuth tokens, or other security mechanisms. +- **Endpoints and resources**: Identify the required API endpoints and resources. Understand + their functions, input parameters, expected outputs, and usage limitations. +- **Data format**: Determine the data format used by the API, such as JSON or XML. This helps + in parsing responses and formatting requests appropriately. +- **Rate Limits and quotas**: Be aware of any rate limits or usage quotas. This information is + crucial to ensure that the integration does not exceed allowed requests or data usage. +- **Error handling**: Learn about error response formats and codes. Knowing this will help in + effectively handling errors and exceptions in your integration. ## Terminology @@ -25,6 +33,12 @@ Before starting the development of a snap-in for integration with an external sy Airdrop extractions are done in _sync runs_. A sync run is one execution of an airdrop or sync. +If you do an initial import from the external system to DevRev, that import will be one sync run. +If you then decide to do a reverse sync from DevRev to the external system, that would be another +sync run. + +Each sync run is comprised out of phases. +Phases follow sequentially, and each can consist of one or more invocations of the snap-in. ### Forward sync @@ -36,17 +50,40 @@ An **extractor** function in the snap-in is responsible for extracting data from A _reverse sync_ is a sync run from DevRev to an external system. It uses a **loader** function, to create or update data in the external system. -### Initial sync +### Initial import + +An _initial import_ is the first import of data from the external system to DevRev. +It is triggered manually by the end user in DevRev's **Airdrops** UI. + +In initial import all data needs to be extracted to create a baseline (while in incremental runs only +updated objects need to be extracted). + +An _initial import_ consists of the following phases: -It is the first forward sync to run. In initial sync all data needs to be extracted to create a baseline (while in incremental runs only updated objects need to be extracted). +1. External sync units extraction +2. Metadata extraction +3. Data extraction +4. Attachments extraction ### 1-way (incremental) sync A _1-way sync_ (or _incremental sync_) refers to any extraction after the initial sync run has been successfully completed. -In the incremental sync an extractor extracts data that was created or updated in the external system after the start +An extractor extracts data that was created or updated in the external system after the start of the latest successful forward sync, including any changes that occurred during the forward sync, but were not picked up by it. +A snap-in must consult its state to get information on when the last successful forward sync started. +Airdrop snap-ins must maintain their own states that persists between phases in a sync run, +as well as between sync runs. + +A 1-way sync consists of the following phases: + +1. Metadata extraction +2. Data extraction +3. Attachments extraction + +A 1-way sync extracts only the domain objects updated or created since the previous successful sync run. + ### Sync unit A _sync unit_ is one connection instance to an external system (for example SalesForce, Freshdesk, Zendesk). diff --git a/fern/docs/pages/airdrop/local-development.mdx b/fern/docs/pages/airdrop/local-development.mdx index 132a8232..7f664e2b 100644 --- a/fern/docs/pages/airdrop/local-development.mdx +++ b/fern/docs/pages/airdrop/local-development.mdx @@ -1,4 +1,4 @@ -For ease of development, you can run your snap-in locally and receive logs directly in your console. +For easier development you can run your snap-in locally and receive logs directly in your console. ## Prerequisites @@ -10,7 +10,7 @@ For ease of development, you can run your snap-in locally and receive logs direc ## Run the template -DevRev provides a template with example code to implement an Airdrop snap-in. +DevRev provides a starter template, which you can run and test out right away. 1. Create a new repository: - Create a new repository from this template by clicking the "Use this template" button in the upper right corner and then "Create a new repository". diff --git a/fern/docs/pages/airdrop/manifest.mdx b/fern/docs/pages/airdrop/manifest.mdx index a0af8ef2..c8039e91 100644 --- a/fern/docs/pages/airdrop/manifest.mdx +++ b/fern/docs/pages/airdrop/manifest.mdx @@ -23,7 +23,7 @@ functions: description: Create blueprint and install initial domain mapping ``` -If loading is not implemented, remove it from the list. This action also removes the option from the UI. +If loading is not implemented, remove it from the list. This action also removes the option from the DevRev app. Check the `imports` section next. diff --git a/fern/docs/pages/airdrop/metadata-extraction.mdx b/fern/docs/pages/airdrop/metadata-extraction.mdx index 46b6b543..19eb9f55 100644 --- a/fern/docs/pages/airdrop/metadata-extraction.mdx +++ b/fern/docs/pages/airdrop/metadata-extraction.mdx @@ -1,29 +1,40 @@ During the metadata extraction phase, the snap-in must provide an `external_domain_metadata.json` file to Airdrop. -This file provides a structured way of describing the external system's domain system, +This file provides a structured way of describing the external system's domain model, including its domain entities, types, relationships, and other metadata. ## Triggering event Airdrop initiates the metadata extraction by starting the snap-in with a message with an event -of type `ExtractionMetadataStart`. +of type `EXTRACTION_METADATA_START`. -The snap-in must respond to Airdrop with a message with an event of type `ExtractionMetadataDone` -when done, or `ExtractionMetadataError` in case of an error. +The snap-in must respond to Airdrop with a message with an event of type `EXTRACTION_METADATA_DONE` +when done, or `EXTRACTION_METADATA_ERROR` in case of an error. ## Response from the snap-in Metadata extraction should be implemented in the [metadata-extraction.ts](https://github.com/devrev/adaas-template/blob/main/code/src/functions/extraction/workers/metadata-extraction.ts) file. ```typescript +import { ExtractorEventType, processTask } from "@devrev/ts-adaas"; import externalDomainMetadata from "../../external-system/external_domain_metadata.json"; -// ... - -await adapter - .getRepo("external_domain_metadata") - ?.push([externalDomainMetadata]); -await adapter.emit(ExtractorEventType.ExtractionMetadataDone); +const repos = [{ itemType: "external_domain_metadata" }]; + +processTask({ + task: async ({ adapter }) => { + adapter.initializeRepos(repos); + await adapter + .getRepo("external_domain_metadata") + ?.push([externalDomainMetadata]); + await adapter.emit(ExtractorEventType.ExtractionMetadataDone); + }, + onTimeout: async ({ adapter }) => { + await adapter.emit(ExtractorEventType.ExtractionMetadataError, { + error: { message: "Failed to extract metadata. Lambda timeout." }, + }); + }, +}); ``` ## Validating metadata @@ -41,12 +52,12 @@ chef-cli validate-metadata < external_domain_metadata.json This outputs any problems there may be with the metadata file. - The detailed format of this metadata is defined by the `external_domain_metadata_schema.json` file. - The file is available as part of the chef-cli package. - There, you can also find an example of a metadata file (`metadata.json`). + The detailed format of this metadata is defined by the + `external_domain_metadata_schema.json` file. The file is available as part of + the chef-cli package. There, you can also find an example of a metadata file + (`metadata.json`). - The transformation can be crafted and finalized further using the chef-cli to ensure extracted data is mapped consistently to the DevRev domain model. @@ -61,6 +72,7 @@ chef-cli infer-metadata example_data_directory > metadata.json To get good results with this approach: 1. Collect example data from the external system and place them in a directory. Each file should: + - Contain the same type of records, named after their type. - Have `.json` or `.jsonl` extension, for example `issues.json`. - Contain either a single JSON array of objects, or newline-separated objects. @@ -70,15 +82,19 @@ To get good results with this approach: 3. Inspect the generated metadata, particularly field types and the suggestions the tool generates. For best results: + - Provide 10-100 examples of each record type (but not more than 1000). - Ensure logically distinct fields are separate keys at the top level. - Use referentially consistent example data if possible. - Make sure IDs are strings, not numbers. -This generated metadata serves as a starting point that needs further refinement. -It can be used to prototype initial domain mappings (by running a sync with it) and to generate example normalized data, but it's important to understand that it's only an initial guess. -The metadata must be carefully reviewed and refined to ensure accuracy and proper representation of your external system's data model. + This generated metadata serves as a starting point that needs further + refinement. It can be used to prototype initial domain mappings (by running a + sync with it) and to generate example normalized data, but it's important to + understand that it's only an initial guess. The metadata must be carefully + reviewed and refined to ensure accuracy and proper representation of your + external system's data model. ## Craft the metadata declaration @@ -112,7 +128,7 @@ be changed by the end user at any time, such as mandatory fields or custom field Although the declaration of record types is arbitrary, they must match the `item_type` field in the artifacts you will upload. - ### Declare the custom record types +### Declare the custom record types If the external system supports custom types, or custom variants of some base record type, and you want to airdrop those too, you have to declare them in the metadata at runtime. @@ -132,7 +148,7 @@ be changed by the end user at any time, such as mandatory fields or custom field } ``` - ### Provide human-readable names to external record types +### Provide human-readable names to external record types Define human-readable names for the record types defined in your metadata file. @@ -155,7 +171,7 @@ be changed by the end user at any time, such as mandatory fields or custom field } ``` - ### Categorize external record types +### Categorize external record types The metadata allows each external record type to be annotated with one category. The category key can be an arbitrary string, but it must match the categories declared under @@ -197,7 +213,7 @@ be changed by the end user at any time, such as mandatory fields or custom field } ``` - ### Declare fields for each record type +### Declare fields for each record type Fields' keys must match what is actually found in the extracted data in the artifacts. Note that field keys are case-sensitive. @@ -207,7 +223,7 @@ be changed by the end user at any time, such as mandatory fields or custom field - `int` - `float` - `text`: Interpreted as plain text. - - `rich_text`: Formatted text with mentions and images. + - `rich_text`: Formatted text with mentions and images. - `reference`: IDs referring to another record. References have to declare what they can refer to, which can be one or more record types (`#record:`) or categories (`#category:`). - `enum`: A string from a predefined set of values with the optional human-readable names for each value. @@ -294,7 +310,7 @@ be changed by the end user at any time, such as mandatory fields or custom field } ``` - ### Declare arrays +### Declare arrays If the field is an array in the extracted data, it is still typed with one of the supported types. Lists must be marked as a `collection`. @@ -314,37 +330,37 @@ be changed by the end user at any time, such as mandatory fields or custom field } } ``` - - - Some references have the role of parent or child. + +- Some references have the role of parent or child. This means that the child record doesn't make sense without its parent, for example a comment attached to a ticket. Assigning a `reference_type` helps Airdrop correctly handle such fields in case the end-user decides to filter some of the parent records out. - - Sometimes the external system uses references besides the primary key of records, for example +- Sometimes the external system uses references besides the primary key of records, for example when referring to a case by serial number, or to a user by their email. To correctly resolve such references, they must be marked with `by_field`, which must be a field existing in that record type, marked `is_identifier`. For example: - ```json {7,16-20} - { - "record_types": { - "users": { - "fields": { - "email": { - "type": "text", - "is_identifier":true - } + +```json {7,16-20} +{ + "record_types": { + "users": { + "fields": { + "email": { + "type": "text", + "is_identifier": true } - }, - "comments": { - "fields": { - "user_email": { - "type": "reference", - "reference": { - "refers_to": { - "#record:users": { - "by_field": "email" - } + } + }, + "comments": { + "fields": { + "user_email": { + "type": "reference", + "reference": { + "refers_to": { + "#record:users": { + "by_field": "email" } } } @@ -352,104 +368,106 @@ be changed by the end user at any time, such as mandatory fields or custom field } } } - ``` - - ### Define field attributes - - External system fields that shouldn't be mapped in reverse should be marked as `is_read_only`. - Depending on their purpose, you can also mark fields as `is_indexed`, `is_identifier`, `is_filterable`, - `is_write_only`, etc. By default, these are set to false. You can find the full list of supported - field attributes and their descriptions in the `external_domain_metadata_schema.json`. - - ### Configure state transitions - - If an external record type has some concept of states, between which only certain transitions are - possible (for example, to move to the `resolved` status, an issue first has to be `in_testing`), you - can declare these in the metadata too. - - This allows creation of a matching *stage diagram* (a collection of stages and their permitted - transitions) in DevRev, which enables a much simpler import and a closer preservation of the external - data than mapping to DevRev's built-in stages. - - This is especially important for two-way sync, as setting the transitions correctly ensures that - the transitions a record undergoes in DevRev can be replicated in the external system. - - To declare this in the metadata, make sure the status is represented as an enum field, and then - declare the allowed transitions (which you might have to retrieve from an API at runtime, if they - are also customized): - - ```json - { - "fields": { - "status": { - "name": "Status", - "is_required": true, - "type": "enum", - "enum": { - "values": [ - { - "key": "detected", - "name": "Detected" - }, - { - "key": "mitigated", - "name": "Mitigated" - }, - { - "key": "rca_ready", - "name": "RCA Ready" - }, - { - "key": "archived", - "name": "Archived" - } - ] - } +} +``` + +### Define field attributes + +External system fields that shouldn't be mapped in reverse should be marked as `is_read_only`. +Depending on their purpose, you can also mark fields as `is_indexed`, `is_identifier`, `is_filterable`, +`is_write_only`, etc. By default, these are set to false. You can find the full list of supported +field attributes and their descriptions in the `external_domain_metadata_schema.json`. + +### Configure state transitions + +If an external record type has some concept of states, between which only certain transitions are +possible (for example, to move to the `resolved` status, an issue first has to be `in_testing`), you +can declare these in the metadata too. + +This allows creation of a matching _stage diagram_ (a collection of stages and their permitted +transitions) in DevRev, which enables a much simpler import and a closer preservation of the external +data than mapping to DevRev's built-in stages. + +This is especially important for two-way sync, as setting the transitions correctly ensures that +the transitions a record undergoes in DevRev can be replicated in the external system. + +To declare this in the metadata, make sure the status is represented as an enum field, and then +declare the allowed transitions (which you might have to retrieve from an API at runtime, if they +are also customized): + +```json +{ + "fields": { + "status": { + "name": "Status", + "is_required": true, + "type": "enum", + "enum": { + "values": [ + { + "key": "detected", + "name": "Detected" + }, + { + "key": "mitigated", + "name": "Mitigated" + }, + { + "key": "rca_ready", + "name": "RCA Ready" + }, + { + "key": "archived", + "name": "Archived" + } + ] + } + } + }, + "stage_diagram": { + "controlling_field": "status", + "starting_stage": "detected", + "all_transitions_allowed": false, + "stages": { + "detected": { + "transitions_to": ["mitigated", "archived", "rca_ready"], + "state": "new" + }, + "mitigated": { + "transitions_to": ["archived", "detected"], + "state": "work_in_progress" + }, + "rca_ready": { + "transitions_to": ["archived"], + "state": "work_in_progress" + }, + "archived": { + "transitions_to": [], + "state": "completed" } }, - "stage_diagram": { - "controlling_field": "status", - "starting_stage": "detected", - "all_transitions_allowed": false, - "stages": { - "detected": { - "transitions_to": ["mitigated", "archived", "rca_ready"], - "state": "new" - }, - "mitigated": { - "transitions_to": ["archived", "detected"], - "state": "work_in_progress" - }, - "rca_ready": { - "transitions_to": ["archived"], - "state": "work_in_progress" - }, - "archived": { - "transitions_to": [], - "state": "completed" - } + "states": { + "new": { + "name": "New" }, - "states": { - "new": { - "name": "New" - }, - "work_in_progress": { - "name": "Work in Progress" - }, - "completed": { - "name": "Completed", - "is_end_state": true - } + "work_in_progress": { + "name": "Work in Progress" + }, + "completed": { + "name": "Completed", + "is_end_state": true } } } - ``` - - In the above example: - - The status field is the controlling field of the stage diagram. - - If a status field has no explicit transitions but you still want a stage diagram, set `all_transitions_allowed` to `true`, which creates a diagram where all the defined stages can transition to each other. - - External systems may categorize statuses (like Jira's status categories), which can be included in the diagram metadata (`states` in the example). - - The `starting_stage` defines the initial stage for new object instances. This data should always be provided if available, otherwise the starting stage is selected alphabetically. - - In current metadata format (v0.2.0), the order and human-readable name are taken from the enum values defined on the controlling field. - - If the `states` field is not provided, default DevRev states are used: `open`, `in_progress`, and `closed`. - +} +``` + +In the above example: + +- The status field is the controlling field of the stage diagram. +- If a status field has no explicit transitions but you still want a stage diagram, set `all_transitions_allowed` to `true`, which creates a diagram where all the defined stages can transition to each other. +- External systems may categorize statuses (like Jira's status categories), which can be included in the diagram metadata (`states` in the example). +- The `starting_stage` defines the initial stage for new object instances. This data should always be provided if available, otherwise the starting stage is selected alphabetically. +- In current metadata format (v0.2.0), the order and human-readable name are taken from the enum values defined on the controlling field. +- If the `states` field is not provided, default DevRev states are used: `open`, `in_progress`, and `closed`. + diff --git a/fern/docs/pages/airdrop/overview.mdx b/fern/docs/pages/airdrop/overview.mdx index 71594983..dd554e7d 100644 --- a/fern/docs/pages/airdrop/overview.mdx +++ b/fern/docs/pages/airdrop/overview.mdx @@ -2,6 +2,9 @@ _Airdrop_ is DevRev’s solution to migrate data. It allows our customers to bri data from external systems to DevRev, export data back to external systems, and keep data in sync between DevRev and the external systems. -DevRev provides a range of snap-ins available through the DevRev marketplace for integration with various external systems like Jira and Zendesk. Sometimes, a specific snap-in may not be available for certain external systems. This is where Airdrop's enhanced functionality comes in, allowing developers to integrate with DevRev’s Airdrop. It enables the creation of external workers (extractors and loaders) to facilitate data transfer from different external systems. +DevRev provides a range of snap-ins available through the DevRev marketplace for integration with +various external systems like Jira and Zendesk. Sometimes, a specific snap-in may not be available +for a given external system. This is why DevRev enables creation of external workers (extractors +and loaders) to other developers, allowing them to integrate with DevRev’s Airdrop and develop their own snap-ins. You can learn more about Airdrop in the [general documentation](https://docs.devrev.ai/import#airdrop-features). diff --git a/fern/docs/pages/airdrop/snap-in-template.mdx b/fern/docs/pages/airdrop/snap-in-template.mdx index a8ef4f43..5af322eb 100644 --- a/fern/docs/pages/airdrop/snap-in-template.mdx +++ b/fern/docs/pages/airdrop/snap-in-template.mdx @@ -1,8 +1,17 @@ -For fast and efficient snap-in development, a template is available at [this GitHub repository](https://github.com/devrev/adaas-template). This template is highly recommended for developers as it streamlines the development process. +For easier snap-in development, a starter template is available [here](https://github.com/devrev/adaas-template). -The template makes use of DevRev's SDK library to manage communication with Airdrop, simplifying the interaction with DevRev's API. It also ensures that the correct communication protocol with Airdrop is followed. +The template makes use of **DevRev's SDK library**, which helps developers build snap-ins that +integrate with DevRev’s Airdrop platform. This library simplifies the workflow for handling +data extraction and loading, event-driven actions, state management, and artifact handling. -Communication between Airdrop and the snap-in involves messages known as events. Depending on the type of event, the snap-in performs different actions. +It provides features such as: + +- **Type definitions**: Structured types for Airdrop control protocol. +- **Event management**: Communication between Airdrop and the snap-in involves messages called + _events_ - depending on the type of event, the snap-in performs different actions. +- **State handling**: Update and access state in real-time within tasks. +- **Artifact management**: Supports batched storage of artifacts. +- **Error & timeout support**: Error handling and timeout management for long-running tasks. ### Snap-in code structure @@ -33,7 +42,7 @@ If you check the template, you can see that a snap-in is a Node.js project with - test/ ``` -The `manifest.yaml` file plays a critical role in a snap-in, holding key configuration details such as the snap-in's name, description, version, connection information, and functionality specifics. This file is the initial step in the development of a snap-in. +The `manifest.yaml` is a configuration file that defines details such as the snap-in's name, description, version, connection information, and functionality specifics. Adjusting this file is the initial step in the development of a snap-in. The `code/` folder holds the implementation of the snap-in. Inside, the `functions/` subfolder includes the logic for retrieving external sync units, extracting data from and loading data to external systems, and mapping between them. diff --git a/fern/versions/public.yml b/fern/versions/public.yml index 4f56cd66..9f4cb559 100644 --- a/fern/versions/public.yml +++ b/fern/versions/public.yml @@ -145,7 +145,7 @@ navigation: - page: "Getting started" slug: getting-started path: ../docs/pages/airdrop/getting-started.mdx - - page: "Airdrop Snap-in template" + - page: "Airdrop snap-in template" slug: snap-in-template path: ../docs/pages/airdrop/snap-in-template.mdx - page: "Local development" From 2e7f6a0b8da807d6a7cda544dc6abf884957cc5f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Patricija=20Bre=C4=8Dko?= Date: Wed, 2 Apr 2025 15:05:51 +0200 Subject: [PATCH 23/27] Fix metadata-extraction file formatting --- fern/docs/pages/airdrop/data-extraction.mdx | 2 +- fern/docs/pages/airdrop/extraction-phases.mdx | 2 +- fern/docs/pages/airdrop/getting-started.mdx | 6 +- .../pages/airdrop/metadata-extraction.mdx | 249 +++++++++--------- 4 files changed, 128 insertions(+), 131 deletions(-) diff --git a/fern/docs/pages/airdrop/data-extraction.mdx b/fern/docs/pages/airdrop/data-extraction.mdx index 1b300463..8cda6698 100644 --- a/fern/docs/pages/airdrop/data-extraction.mdx +++ b/fern/docs/pages/airdrop/data-extraction.mdx @@ -101,7 +101,7 @@ Effective use of the state and breaking down the problem into smaller chunks are The snap-in starter template contains an [example](https://github.com/devrev/adaas-template/blob/main/code/src/functions/extraction/index.ts) of a simple state. Adding more data to the state can help with pagination and rate limiting by saving the point at which extraction was left off. -To more easily test the state in development, you can decrease the timeout between snap-in invocations. +To test the state in development, you can decrease the timeout between snap-in invocations. ```typescript await spawn({ diff --git a/fern/docs/pages/airdrop/extraction-phases.mdx b/fern/docs/pages/airdrop/extraction-phases.mdx index 51ef5f6b..46ee7756 100644 --- a/fern/docs/pages/airdrop/extraction-phases.mdx +++ b/fern/docs/pages/airdrop/extraction-phases.mdx @@ -1,4 +1,4 @@ -Each snap-in must handle all the phases of Airdrop extraction. In a snap-in,you typically define a run +Each snap-in must handle all the phases of Airdrop extraction. In a snap-in, you typically define a run function that iterates over events and invokes workers per extraction phase. The SDK library exports `processTask` to structure the work within each phase, and `onTimeout` function diff --git a/fern/docs/pages/airdrop/getting-started.mdx b/fern/docs/pages/airdrop/getting-started.mdx index e60b140d..6219731d 100644 --- a/fern/docs/pages/airdrop/getting-started.mdx +++ b/fern/docs/pages/airdrop/getting-started.mdx @@ -15,7 +15,7 @@ Before starting the development of a snap-in for integration with an external sy consider gathering the following information: - **API documentation**: Obtain the official API documentation of the external system. - This will be the primary source of information about how to connect and interact with the system. + This is the primary source of information about how to connect and interact with the system. - **Authentication and authorization**: Understand the authentication and authorization methods required. This may include API keys, OAuth tokens, or other security mechanisms. - **Endpoints and resources**: Identify the required API endpoints and resources. Understand @@ -24,8 +24,8 @@ consider gathering the following information: in parsing responses and formatting requests appropriately. - **Rate Limits and quotas**: Be aware of any rate limits or usage quotas. This information is crucial to ensure that the integration does not exceed allowed requests or data usage. -- **Error handling**: Learn about error response formats and codes. Knowing this will help in - effectively handling errors and exceptions in your integration. +- **Error handling**: Learn about error response formats and codes. Knowing this helps in + handling errors and exceptions in your integration. ## Terminology diff --git a/fern/docs/pages/airdrop/metadata-extraction.mdx b/fern/docs/pages/airdrop/metadata-extraction.mdx index 19eb9f55..8b0e7822 100644 --- a/fern/docs/pages/airdrop/metadata-extraction.mdx +++ b/fern/docs/pages/airdrop/metadata-extraction.mdx @@ -107,7 +107,6 @@ Metadata declarations include both _static declarations_, formulated by deductio external domain system and DevRev domain system, and _dynamic declarations_ that are obtained during a snap-in run from external system APIs (since they are configurable in the external system and can be changed by the end user at any time, such as mandatory fields or custom fields). - ### Declare the extracted record types @@ -128,7 +127,7 @@ be changed by the end user at any time, such as mandatory fields or custom field Although the declaration of record types is arbitrary, they must match the `item_type` field in the artifacts you will upload. -### Declare the custom record types + ### Declare the custom record types If the external system supports custom types, or custom variants of some base record type, and you want to airdrop those too, you have to declare them in the metadata at runtime. @@ -148,7 +147,7 @@ be changed by the end user at any time, such as mandatory fields or custom field } ``` -### Provide human-readable names to external record types + ### Provide human-readable names to external record types Define human-readable names for the record types defined in your metadata file. @@ -171,7 +170,7 @@ be changed by the end user at any time, such as mandatory fields or custom field } ``` -### Categorize external record types + ### Categorize external record types The metadata allows each external record type to be annotated with one category. The category key can be an arbitrary string, but it must match the categories declared under @@ -213,7 +212,7 @@ be changed by the end user at any time, such as mandatory fields or custom field } ``` -### Declare fields for each record type + ### Declare fields for each record type Fields' keys must match what is actually found in the extracted data in the artifacts. Note that field keys are case-sensitive. @@ -223,7 +222,7 @@ be changed by the end user at any time, such as mandatory fields or custom field - `int` - `float` - `text`: Interpreted as plain text. - - `rich_text`: Formatted text with mentions and images. + - `rich_text`: Formatted text with mentions and images. - `reference`: IDs referring to another record. References have to declare what they can refer to, which can be one or more record types (`#record:`) or categories (`#category:`). - `enum`: A string from a predefined set of values with the optional human-readable names for each value. @@ -310,7 +309,7 @@ be changed by the end user at any time, such as mandatory fields or custom field } ``` -### Declare arrays + ### Declare arrays If the field is an array in the extracted data, it is still typed with one of the supported types. Lists must be marked as a `collection`. @@ -330,37 +329,37 @@ be changed by the end user at any time, such as mandatory fields or custom field } } ``` - -- Some references have the role of parent or child. + + - Some references have the role of parent or child. This means that the child record doesn't make sense without its parent, for example a comment attached to a ticket. Assigning a `reference_type` helps Airdrop correctly handle such fields in case the end-user decides to filter some of the parent records out. -- Sometimes the external system uses references besides the primary key of records, for example + - Sometimes the external system uses references besides the primary key of records, for example when referring to a case by serial number, or to a user by their email. To correctly resolve such references, they must be marked with `by_field`, which must be a field existing in that record type, marked `is_identifier`. For example: - -```json {7,16-20} -{ - "record_types": { - "users": { - "fields": { - "email": { - "type": "text", - "is_identifier": true + ```json {7,16-20} + { + "record_types": { + "users": { + "fields": { + "email": { + "type": "text", + "is_identifier":true + } } - } - }, - "comments": { - "fields": { - "user_email": { - "type": "reference", - "reference": { - "refers_to": { - "#record:users": { - "by_field": "email" + }, + "comments": { + "fields": { + "user_email": { + "type": "reference", + "reference": { + "refers_to": { + "#record:users": { + "by_field": "email" + } } } } @@ -368,106 +367,104 @@ be changed by the end user at any time, such as mandatory fields or custom field } } } -} -``` - -### Define field attributes - -External system fields that shouldn't be mapped in reverse should be marked as `is_read_only`. -Depending on their purpose, you can also mark fields as `is_indexed`, `is_identifier`, `is_filterable`, -`is_write_only`, etc. By default, these are set to false. You can find the full list of supported -field attributes and their descriptions in the `external_domain_metadata_schema.json`. - -### Configure state transitions - -If an external record type has some concept of states, between which only certain transitions are -possible (for example, to move to the `resolved` status, an issue first has to be `in_testing`), you -can declare these in the metadata too. - -This allows creation of a matching _stage diagram_ (a collection of stages and their permitted -transitions) in DevRev, which enables a much simpler import and a closer preservation of the external -data than mapping to DevRev's built-in stages. - -This is especially important for two-way sync, as setting the transitions correctly ensures that -the transitions a record undergoes in DevRev can be replicated in the external system. - -To declare this in the metadata, make sure the status is represented as an enum field, and then -declare the allowed transitions (which you might have to retrieve from an API at runtime, if they -are also customized): - -```json -{ - "fields": { - "status": { - "name": "Status", - "is_required": true, - "type": "enum", - "enum": { - "values": [ - { - "key": "detected", - "name": "Detected" - }, - { - "key": "mitigated", - "name": "Mitigated" - }, - { - "key": "rca_ready", - "name": "RCA Ready" - }, - { - "key": "archived", - "name": "Archived" - } - ] - } - } - }, - "stage_diagram": { - "controlling_field": "status", - "starting_stage": "detected", - "all_transitions_allowed": false, - "stages": { - "detected": { - "transitions_to": ["mitigated", "archived", "rca_ready"], - "state": "new" - }, - "mitigated": { - "transitions_to": ["archived", "detected"], - "state": "work_in_progress" - }, - "rca_ready": { - "transitions_to": ["archived"], - "state": "work_in_progress" - }, - "archived": { - "transitions_to": [], - "state": "completed" + ``` + + ### Define field attributes + + External system fields that shouldn't be mapped in reverse should be marked as `is_read_only`. + Depending on their purpose, you can also mark fields as `is_indexed`, `is_identifier`, `is_filterable`, + `is_write_only`, etc. By default, these are set to false. You can find the full list of supported + field attributes and their descriptions in the `external_domain_metadata_schema.json`. + + ### Configure state transitions + + If an external record type has some concept of states, between which only certain transitions are + possible (for example, to move to the `resolved` status, an issue first has to be `in_testing`), you + can declare these in the metadata too. + + This allows creation of a matching *stage diagram* (a collection of stages and their permitted + transitions) in DevRev, which enables a much simpler import and a closer preservation of the external + data than mapping to DevRev's built-in stages. + + This is especially important for two-way sync, as setting the transitions correctly ensures that + the transitions a record undergoes in DevRev can be replicated in the external system. + + To declare this in the metadata, make sure the status is represented as an enum field, and then + declare the allowed transitions (which you might have to retrieve from an API at runtime, if they + are also customized): + + ```json + { + "fields": { + "status": { + "name": "Status", + "is_required": true, + "type": "enum", + "enum": { + "values": [ + { + "key": "detected", + "name": "Detected" + }, + { + "key": "mitigated", + "name": "Mitigated" + }, + { + "key": "rca_ready", + "name": "RCA Ready" + }, + { + "key": "archived", + "name": "Archived" + } + ] + } } }, - "states": { - "new": { - "name": "New" - }, - "work_in_progress": { - "name": "Work in Progress" + "stage_diagram": { + "controlling_field": "status", + "starting_stage": "detected", + "all_transitions_allowed": false, + "stages": { + "detected": { + "transitions_to": ["mitigated", "archived", "rca_ready"], + "state": "new" + }, + "mitigated": { + "transitions_to": ["archived", "detected"], + "state": "work_in_progress" + }, + "rca_ready": { + "transitions_to": ["archived"], + "state": "work_in_progress" + }, + "archived": { + "transitions_to": [], + "state": "completed" + } }, - "completed": { - "name": "Completed", - "is_end_state": true + "states": { + "new": { + "name": "New" + }, + "work_in_progress": { + "name": "Work in Progress" + }, + "completed": { + "name": "Completed", + "is_end_state": true + } } } } -} -``` - -In the above example: - -- The status field is the controlling field of the stage diagram. -- If a status field has no explicit transitions but you still want a stage diagram, set `all_transitions_allowed` to `true`, which creates a diagram where all the defined stages can transition to each other. -- External systems may categorize statuses (like Jira's status categories), which can be included in the diagram metadata (`states` in the example). -- The `starting_stage` defines the initial stage for new object instances. This data should always be provided if available, otherwise the starting stage is selected alphabetically. -- In current metadata format (v0.2.0), the order and human-readable name are taken from the enum values defined on the controlling field. -- If the `states` field is not provided, default DevRev states are used: `open`, `in_progress`, and `closed`. - + ``` + + In the above example: + - The status field is the controlling field of the stage diagram. + - If a status field has no explicit transitions but you still want a stage diagram, set `all_transitions_allowed` to `true`, which creates a diagram where all the defined stages can transition to each other. + - External systems may categorize statuses (like Jira's status categories), which can be included in the diagram metadata (`states` in the example). + - The `starting_stage` defines the initial stage for new object instances. This data should always be provided if available, otherwise the starting stage is selected alphabetically. + - In current metadata format (v0.2.0), the order and human-readable name are taken from the enum values defined on the controlling field. + - If the `states` field is not provided, default DevRev states are used: `open`, `in_progress`, and `closed`. + \ No newline at end of file From 95f4a2cb00436f2f9917960dc556b8b21c777b90 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Patricija=20Bre=C4=8Dko?= Date: Thu, 3 Apr 2025 12:31:11 +0200 Subject: [PATCH 24/27] Minor text changes --- .../pages/airdrop/attachments-extraction.mdx | 4 +- fern/docs/pages/airdrop/data-extraction.mdx | 4 +- fern/docs/pages/airdrop/extraction-phases.mdx | 27 ++++--- fern/docs/pages/airdrop/getting-started.mdx | 10 +-- fern/docs/pages/airdrop/keyring.mdx | 64 --------------- fern/docs/pages/airdrop/manifest.mdx | 67 +++++++++++++++- .../pages/airdrop/metadata-extraction.mdx | 2 +- fern/versions/public.yml | 78 +++++++++---------- 8 files changed, 132 insertions(+), 124 deletions(-) delete mode 100644 fern/docs/pages/airdrop/keyring.mdx diff --git a/fern/docs/pages/airdrop/attachments-extraction.mdx b/fern/docs/pages/airdrop/attachments-extraction.mdx index 85519ee1..800f3f0d 100644 --- a/fern/docs/pages/airdrop/attachments-extraction.mdx +++ b/fern/docs/pages/airdrop/attachments-extraction.mdx @@ -28,7 +28,9 @@ with an event of type `EXTRACTION_ATTACHMENTS_DONE`. If attachment extraction fails the snap-in must respond to Airdrop with a message with an event of type `EXTRACTION_ATTACHMENTS_ERROR`. -## Response from the snap-in +## Implementation + +Attachments extraction should be implemented in the [attachments-extraction.ts](https://github.com/devrev/adaas-template/blob/main/code/src/functions/extraction/workers/attachments-extraction.ts) file. After uploading an attachment or a batch of attachments, the extractor also has to prepare and upload a file specifying the extracted and uploaded attachments. diff --git a/fern/docs/pages/airdrop/data-extraction.mdx b/fern/docs/pages/airdrop/data-extraction.mdx index 8cda6698..7ebab6dd 100644 --- a/fern/docs/pages/airdrop/data-extraction.mdx +++ b/fern/docs/pages/airdrop/data-extraction.mdx @@ -32,9 +32,9 @@ type `EXTRACTION_DATA_DONE`. If data extraction failed in any moment of extraction, the snap-in must respond to Airdrop with a message with event of type `EXTRACTION_DATA_ERROR`. -Data extraction should be implemented in the [data-extraction.ts](https://github.com/devrev/adaas-template/blob/main/code/src/functions/extraction/workers/data-extraction.ts) file. +## Implementation -## Response from the snap-in +Data extraction should be implemented in the [data-extraction.ts](https://github.com/devrev/adaas-template/blob/main/code/src/functions/extraction/workers/data-extraction.ts) file. During the data extraction phase, the snap-in uploads batches of extracted items (with tooling from `@devrev/adaas-sdk`). diff --git a/fern/docs/pages/airdrop/extraction-phases.mdx b/fern/docs/pages/airdrop/extraction-phases.mdx index 46ee7756..0023961e 100644 --- a/fern/docs/pages/airdrop/extraction-phases.mdx +++ b/fern/docs/pages/airdrop/extraction-phases.mdx @@ -19,14 +19,14 @@ The state is stored in the adapter and can be retrieved using the `adapter.state ```typescript import { AirdropEvent, EventType, spawn } from "@devrev/ts-adaas"; -interface DummyExtractorState { - issues: { completed: boolean }; +export interface ExtractorState { + todos: { completed: boolean }; users: { completed: boolean }; attachments: { completed: boolean }; } -const initialState: DummyExtractorState = { - issues: { completed: false }, +export const initialState: ExtractorState = { + todos: { completed: false }, users: { completed: false }, attachments: { completed: false }, }; @@ -35,14 +35,18 @@ function getWorkerPerExtractionPhase(event: AirdropEvent) { let path; switch (event.payload.event_type) { case EventType.ExtractionExternalSyncUnitsStart: - path = __dirname + "/workers/external-sync-units-extraction"; + path = __dirname + '/workers/external-sync-units-extraction'; break; case EventType.ExtractionMetadataStart: - path = __dirname + "/workers/metadata-extraction"; + path = __dirname + '/workers/metadata-extraction'; break; case EventType.ExtractionDataStart: case EventType.ExtractionDataContinue: - path = __dirname + "/workers/data-extraction"; + path = __dirname + '/workers/data-extraction'; + break; + case EventType.ExtractionAttachmentsStart: + case EventType.ExtractionAttachmentsContinue: + path = __dirname + '/workers/attachments-extraction'; break; } return path; @@ -51,13 +55,14 @@ function getWorkerPerExtractionPhase(event: AirdropEvent) { const run = async (events: AirdropEvent[]) => { for (const event of events) { const file = getWorkerPerExtractionPhase(event); - await spawn({ + await spawn({ event, initialState, workerPath: file, - options: { - isLocalDevelopment: true, - }, + + // TODO: If needed you can pass additional options to the spawn function. + // For example timeout of the lambda, batch size, etc. + // options: {}, }); } }; diff --git a/fern/docs/pages/airdrop/getting-started.mdx b/fern/docs/pages/airdrop/getting-started.mdx index 6219731d..0cc9e8fc 100644 --- a/fern/docs/pages/airdrop/getting-started.mdx +++ b/fern/docs/pages/airdrop/getting-started.mdx @@ -22,13 +22,17 @@ consider gathering the following information: their functions, input parameters, expected outputs, and usage limitations. - **Data format**: Determine the data format used by the API, such as JSON or XML. This helps in parsing responses and formatting requests appropriately. -- **Rate Limits and quotas**: Be aware of any rate limits or usage quotas. This information is +- **Rate limits and quotas**: Be aware of any rate limits or usage quotas. This information is crucial to ensure that the integration does not exceed allowed requests or data usage. - **Error handling**: Learn about error response formats and codes. Knowing this helps in handling errors and exceptions in your integration. ## Terminology +### Sync unit + +A _sync unit_ is one connection instance to an external system (for example SalesForce, Freshdesk, Zendesk). + ### Sync run Airdrop extractions are done in _sync runs_. @@ -83,7 +87,3 @@ A 1-way sync consists of the following phases: 3. Attachments extraction A 1-way sync extracts only the domain objects updated or created since the previous successful sync run. - -### Sync unit - -A _sync unit_ is one connection instance to an external system (for example SalesForce, Freshdesk, Zendesk). diff --git a/fern/docs/pages/airdrop/keyring.mdx b/fern/docs/pages/airdrop/keyring.mdx deleted file mode 100644 index b2c059ef..00000000 --- a/fern/docs/pages/airdrop/keyring.mdx +++ /dev/null @@ -1,64 +0,0 @@ -_Keyrings_ are a collection of authentication information, used by a snap-in to authenticate to the external system in API calls. This can include a key (for example, a PAT token or API key), its type, the organization ID for which a key is valid, and in some cases the organization name. - -Keyrings provide a secure way to store and manage credentials within your DevRev snap-in. -This eliminates the need to expose sensitive information like passwords or access tokens directly -within your code or configuration files, enhancing overall security. -They also provide a valid token by abstracting OAuth token renewal from the end user. - -They are called **Connections** in the DevRev app. - -### Configure a keyring - -Keyrings are configured in the `manifest.yaml` by configuring a `keyring_type`, like in the [example](https://github.com/devrev/adaas-template/blob/main/manifest.yaml). - -```yaml -keyring_types: - - id: - name: - description: - # The kind field specifies the type of keyring. - kind: <"secret"/"oauth2"> - # is_subdomain field specifies whether the keyring contains a subdomain. - # Enabling this field allows the keyring to get the subdomain from the user during creation. - # This is useful when the keyring requires a subdomain as part of the configuration. - # Default is false. - is_subdomain: - # Name of the external system you are importing from. - # This system name should start with a capital letter. - external_system_name: - secret_config: - # Define the fields in the secret. - # Each element represents one input field. - # User will provide this information when creating the connection in the UI. - # If omitted, the user will be asked for a generic secret. - fields: - - id: - name: - description: - is_optional: # optional: whether the field is optional or not. Default is false. - # The token_verification section is used to verify the token provided by the user. - token_verification: - # The URL to which the token will be sent for verification. - url: - # The HTTP method to be used for the verification request. - method: # The HTTP method to be used for the verification request. - # Optional: headers to be included in the verification request. - headers: - : - # Optional: query parameters to be included in the verification request. - query_params: - : # optional: query parameters to be included in the verification request. - # Fetching Organization Data: This allows you to retrieve additional information about the user's organization. - organization_data: - type: "config" - # The URL to which the request is sent to fetch organization data. - url: - # The HTTP method used to send the request. - method: "GET" - headers: - : - # The jq filter used to extract the organization data from the response. - response_jq: -``` - -You can find more information about keyrings and keyring types [here](/snapin-development/references/keyrings/keyring-intro). diff --git a/fern/docs/pages/airdrop/manifest.mdx b/fern/docs/pages/airdrop/manifest.mdx index c8039e91..e3e3482a 100644 --- a/fern/docs/pages/airdrop/manifest.mdx +++ b/fern/docs/pages/airdrop/manifest.mdx @@ -44,4 +44,69 @@ If loading is not used, remove `loader_function`. Ensure that `extractor_function` and `loader_function` names correspond with those in the `functions` section. -Proceed to the next section, which dives in on establishing a connection to the external system using `keyring_types`. +## Establish a connection to the external system + +_Keyrings_ are a collection of authentication information, used by a snap-in to authenticate to the external system in API calls. This can include a key (for example, a PAT token or API key), its type, the organization ID for which a key is valid, and in some cases the organization name. + +Keyrings provide a secure way to store and manage credentials within your DevRev snap-in. +This eliminates the need to expose sensitive information like passwords or access tokens directly +within your code or configuration files, enhancing overall security. +They also provide a valid token by abstracting OAuth token renewal from the end user. + +They are called **Connections** in the DevRev app. + +### Configure a keyring + +Keyrings are configured in the `manifest.yaml` by configuring a `keyring_type`, like in the [example](https://github.com/devrev/adaas-template/blob/main/manifest.yaml). + +```yaml +keyring_types: + - id: + name: + description: + # The kind field specifies the type of keyring. + kind: <"secret"/"oauth2"> + # is_subdomain field specifies whether the keyring contains a subdomain. + # Enabling this field allows the keyring to get the subdomain from the user during creation. + # This is useful when the keyring requires a subdomain as part of the configuration. + # Default is false. + is_subdomain: + # Name of the external system you are importing from. + # This system name should start with a capital letter. + external_system_name: + secret_config: + # Define the fields in the secret. + # Each element represents one input field. + # User will provide this information when creating the connection in the UI. + # If omitted, the user will be asked for a generic secret. + fields: + - id: + name: + description: + is_optional: # optional: whether the field is optional or not. Default is false. + # The token_verification section is used to verify the token provided by the user. + token_verification: + # The URL to which the token will be sent for verification. + url: + # The HTTP method to be used for the verification request. + method: # The HTTP method to be used for the verification request. + # Optional: headers to be included in the verification request. + headers: + : + # Optional: query parameters to be included in the verification request. + query_params: + : # optional: query parameters to be included in the verification request. + # Fetching Organization Data: This allows you to retrieve additional information about the user's organization. + organization_data: + type: "config" + # The URL to which the request is sent to fetch organization data. + url: + # The HTTP method used to send the request. + method: "GET" + headers: + : + # The jq filter used to extract the organization data from the response. + response_jq: +``` + +You can find more information about keyrings and keyring types [here](/snapin-development/references/keyrings/keyring-intro). \ No newline at end of file diff --git a/fern/docs/pages/airdrop/metadata-extraction.mdx b/fern/docs/pages/airdrop/metadata-extraction.mdx index 8b0e7822..cbdfd22d 100644 --- a/fern/docs/pages/airdrop/metadata-extraction.mdx +++ b/fern/docs/pages/airdrop/metadata-extraction.mdx @@ -11,7 +11,7 @@ of type `EXTRACTION_METADATA_START`. The snap-in must respond to Airdrop with a message with an event of type `EXTRACTION_METADATA_DONE` when done, or `EXTRACTION_METADATA_ERROR` in case of an error. -## Response from the snap-in +## Implementation Metadata extraction should be implemented in the [metadata-extraction.ts](https://github.com/devrev/adaas-template/blob/main/code/src/functions/extraction/workers/metadata-extraction.ts) file. diff --git a/fern/versions/public.yml b/fern/versions/public.yml index d1609d44..013929a8 100644 --- a/fern/versions/public.yml +++ b/fern/versions/public.yml @@ -144,45 +144,45 @@ navigation: - page: "Getting started" slug: getting-started path: ../docs/pages/airdrop/getting-started.mdx - - page: "Airdrop snap-in template" - slug: snap-in-template - path: ../docs/pages/airdrop/snap-in-template.mdx - - page: "Local development" - slug: local-development - path: ../docs/pages/airdrop/local-development.mdx - - page: "Manifest" - slug: manifest - path: ../docs/pages/airdrop/manifest.mdx - - page: "Keyring" - slug: keyring - path: ../docs/pages/airdrop/keyring.mdx - - page: "Extraction phases" - slug: extraction-phases - path: ../docs/pages/airdrop/extraction-phases.mdx - - page: "External sync units extraction" - slug: external-sync-units-extraction - path: ../docs/pages/airdrop/external-sync-units-extraction.mdx - - page: "Metadata extraction" - slug: metadata-extraction - path: ../docs/pages/airdrop/metadata-extraction.mdx - - page: "Data extraction" - slug: data-extraction - path: ../docs/pages/airdrop/data-extraction.mdx - - page: "Initial domain mapping" - hidden: false - slug: initial-domain-mapping - path: ../docs/pages/airdrop/initial-domain-mapping.mdx - - page: "Attachments extraction" - slug: extract-attachments - path: ../docs/pages/airdrop/attachments-extraction.mdx - - page: "Loading phases" - slug: loading-phases - hidden: true - path: ../docs/pages/airdrop/loading-phases.mdx - - page: "Data and attachments deletion" - slug: data-attachments-deletion - hidden: false - path: ../docs/pages/airdrop/data-attachments-deletion.mdx + - page: "Development guide" + skip-slug: true + contents: + - page: "Airdrop snap-in template" + slug: snap-in-template + path: ../docs/pages/airdrop/snap-in-template.mdx + - page: "Local development" + slug: local-development + path: ../docs/pages/airdrop/local-development.mdx + - page: "Manifest" + slug: manifest + path: ../docs/pages/airdrop/manifest.mdx + - page: "Extraction phases" + slug: extraction-phases + path: ../docs/pages/airdrop/extraction-phases.mdx + - page: "External sync units extraction" + slug: external-sync-units-extraction + path: ../docs/pages/airdrop/external-sync-units-extraction.mdx + - page: "Metadata extraction" + slug: metadata-extraction + path: ../docs/pages/airdrop/metadata-extraction.mdx + - page: "Initial domain mapping" + hidden: false + slug: initial-domain-mapping + path: ../docs/pages/airdrop/initial-domain-mapping.mdx + - page: "Data extraction" + slug: data-extraction + path: ../docs/pages/airdrop/data-extraction.mdx + - page: "Attachments extraction" + slug: extract-attachments + path: ../docs/pages/airdrop/attachments-extraction.mdx + - page: "Loading phases" + slug: loading-phases + hidden: true + path: ../docs/pages/airdrop/loading-phases.mdx + - page: "Data and attachments deletion" + slug: data-attachments-deletion + hidden: true + path: ../docs/pages/airdrop/data-attachments-deletion.mdx - page: "Examples of snap-ins" slug: snap-in-examples path: ../docs/pages/airdrop/snap-in-examples.mdx From 568b2a33bdf76dbdb141a63a68ab6fc248898236 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Patricija=20Bre=C4=8Dko?= Date: Thu, 3 Apr 2025 12:52:14 +0200 Subject: [PATCH 25/27] Fix public.yml --- fern/versions/public.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/fern/versions/public.yml b/fern/versions/public.yml index 013929a8..2160a5c9 100644 --- a/fern/versions/public.yml +++ b/fern/versions/public.yml @@ -145,7 +145,6 @@ navigation: slug: getting-started path: ../docs/pages/airdrop/getting-started.mdx - page: "Development guide" - skip-slug: true contents: - page: "Airdrop snap-in template" slug: snap-in-template From 1e6cdff9398adc10f9ef0f8054c6f0eef38a7831 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Patricija=20Bre=C4=8Dko?= Date: Thu, 3 Apr 2025 12:56:01 +0200 Subject: [PATCH 26/27] Fix public.yml again --- fern/versions/public.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/fern/versions/public.yml b/fern/versions/public.yml index 2160a5c9..b94354e7 100644 --- a/fern/versions/public.yml +++ b/fern/versions/public.yml @@ -144,7 +144,8 @@ navigation: - page: "Getting started" slug: getting-started path: ../docs/pages/airdrop/getting-started.mdx - - page: "Development guide" + - section: "Development guide" + skip-slug: true contents: - page: "Airdrop snap-in template" slug: snap-in-template From abb979e4ca3c711e361aad4f12932070e3b5d097 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Patricija=20Bre=C4=8Dko?= Date: Fri, 4 Apr 2025 10:39:36 +0200 Subject: [PATCH 27/27] Some fixes and add supported object types --- .../pages/airdrop/attachments-extraction.mdx | 5 +- fern/docs/pages/airdrop/data-extraction.mdx | 6 +- .../airdrop/extraction-getting-started.mdx | 15 - fern/docs/pages/airdrop/extraction-phases.mdx | 6 +- fern/docs/pages/airdrop/getting-started.mdx | 13 +- fern/docs/pages/airdrop/loading-data.mdx | 1 - fern/docs/pages/airdrop/local-development.mdx | 7 +- fern/docs/pages/airdrop/manifest.mdx | 10 +- .../pages/airdrop/metadata-extraction.mdx | 2 + fern/docs/pages/airdrop/overview.mdx | 5 +- .../pages/airdrop/publish-to-marketplace.mdx | 6 +- fern/docs/pages/airdrop/snap-in-template.mdx | 15 +- .../pages/airdrop/supported-object-types.mdx | 1147 +++++++++++++++++ fern/versions/public.yml | 3 + 14 files changed, 1194 insertions(+), 47 deletions(-) delete mode 100644 fern/docs/pages/airdrop/extraction-getting-started.mdx delete mode 100644 fern/docs/pages/airdrop/loading-data.mdx create mode 100644 fern/docs/pages/airdrop/supported-object-types.mdx diff --git a/fern/docs/pages/airdrop/attachments-extraction.mdx b/fern/docs/pages/airdrop/attachments-extraction.mdx index 800f3f0d..6e20a539 100644 --- a/fern/docs/pages/airdrop/attachments-extraction.mdx +++ b/fern/docs/pages/airdrop/attachments-extraction.mdx @@ -1,4 +1,4 @@ -For the attachment extraction phase of the import process, the extractor has to upload each attachment to DevRev and associate it with the data object it is attached to. +In the attachment extraction phase, the snap-in has to upload each attachment to DevRev and associate it with its parent data object. ## Triggering event @@ -30,7 +30,8 @@ type `EXTRACTION_ATTACHMENTS_ERROR`. ## Implementation -Attachments extraction should be implemented in the [attachments-extraction.ts](https://github.com/devrev/adaas-template/blob/main/code/src/functions/extraction/workers/attachments-extraction.ts) file. +Attachments extraction is already provided by SDK, but if you need to customize it for your use case, +it should be implemented in the [attachments-extraction.ts](https://github.com/devrev/adaas-template/blob/main/code/src/functions/extraction/workers/attachments-extraction.ts) file. After uploading an attachment or a batch of attachments, the extractor also has to prepare and upload a file specifying the extracted and uploaded attachments. diff --git a/fern/docs/pages/airdrop/data-extraction.mdx b/fern/docs/pages/airdrop/data-extraction.mdx index 7ebab6dd..336ec3d3 100644 --- a/fern/docs/pages/airdrop/data-extraction.mdx +++ b/fern/docs/pages/airdrop/data-extraction.mdx @@ -88,7 +88,11 @@ echo '{}' | chef-cli fuzz-extracted -r issue -m external_domain_metadata.json > ## State handling -Since each snap-in invocation is a separate runtime instance (with a maximum execution time of 12 minutes), it does not know what has been previously accomplished or how many records have already been extracted. To enable information passing between invocations and runs, support has been added for saving a limited amount of data as the snap-in `state`. Snap-in state persists between phases in one sync run as well as between multiple sync runs. +Since each snap-in invocation is a separate runtime instance (with a maximum execution time of 12 minutes), +it does not know what has been previously accomplished or how many records have already been extracted. +To enable information passing between invocations and runs, support has been added for saving a limited amount +of data as the snap-in `state`. Snap-in `state` persists between phases in one sync run as well as between multiple sync runs. +You can access the `state` through SDK's `adapter` object. A snap-in must consult its state to obtain information on when the last successful forward sync started. diff --git a/fern/docs/pages/airdrop/extraction-getting-started.mdx b/fern/docs/pages/airdrop/extraction-getting-started.mdx deleted file mode 100644 index 790a2e84..00000000 --- a/fern/docs/pages/airdrop/extraction-getting-started.mdx +++ /dev/null @@ -1,15 +0,0 @@ -Extraction generally involves four sequential phases: - -1. external sync units extraction (only on initial import), -2. metadata extraction, -3. data extraction, -4. attachments extraction. - -### Before you start implementation - -It is recommended you consider the following points before you start the implementation: - -1. Examine the domain model of the external system and identify available and required fields for its objects. -2. Examine DevRev's domain model - use the specification given [here](https://github.com/devrev/airdrop-recipe-manager/blob/main/v2/devrev_metadata/devrev_metadata_docs.md). -3. Decide how objects should correspond to each other and plan field mappings accordingly. Consider how to handle required fields in one system that are optional in the other. -4. Check the external system API too see what are options and limitations. diff --git a/fern/docs/pages/airdrop/extraction-phases.mdx b/fern/docs/pages/airdrop/extraction-phases.mdx index 0023961e..75f4e26d 100644 --- a/fern/docs/pages/airdrop/extraction-phases.mdx +++ b/fern/docs/pages/airdrop/extraction-phases.mdx @@ -4,9 +4,9 @@ function that iterates over events and invokes workers per extraction phase. The SDK library exports `processTask` to structure the work within each phase, and `onTimeout` function to handle timeouts. -The snap-in extraction lifecycle consists of three main phases: External Sync Units Extraction, -Metadata Extraction, and Data Extraction. Each phase is defined in a separate file and is responsible -for fetching the respective data. +The Airdrop snap-in extraction lifecycle consists of four phases: External Sync Units Extraction, +Metadata Extraction, Data Extraction and Attachments Extraction. Each phase is defined in a +separate file and is responsible for fetching the respective data. The SDK library provides a repository management system to handle artifacts in batches. The `initializeRepos` function initializes the repositories, and the `push` function uploads the diff --git a/fern/docs/pages/airdrop/getting-started.mdx b/fern/docs/pages/airdrop/getting-started.mdx index 0cc9e8fc..ddca916d 100644 --- a/fern/docs/pages/airdrop/getting-started.mdx +++ b/fern/docs/pages/airdrop/getting-started.mdx @@ -1,3 +1,6 @@ +Before starting snap-in development, it's important to understand these fundamental terms and concepts, +as well as the capabilities and limitations of the external system being integrated. + ## DevRev organization If this is your first time setting up an organization within DevRev, you need to create a @@ -31,14 +34,18 @@ consider gathering the following information: ### Sync unit -A _sync unit_ is one connection instance to an external system (for example SalesForce, Freshdesk, Zendesk). +A _sync unit_ is one self encompassing unit of data that is synced to an external system. Examples: +- Jira project +- SalesForce +- Zendesk ### Sync run Airdrop extractions are done in _sync runs_. -A sync run is one execution of an airdrop or sync. +A sync run is one end-to-end (extract-transform-load) execution of a _sync unit_. If you do an initial import from the external system to DevRev, that import will be one sync run. -If you then decide to do a reverse sync from DevRev to the external system, that would be another +Another import in the same direction is another sync run. +And if you then decide to do a reverse sync from DevRev to the external system, that would be another sync run. Each sync run is comprised out of phases. diff --git a/fern/docs/pages/airdrop/loading-data.mdx b/fern/docs/pages/airdrop/loading-data.mdx deleted file mode 100644 index 1333ed77..00000000 --- a/fern/docs/pages/airdrop/loading-data.mdx +++ /dev/null @@ -1 +0,0 @@ -TODO diff --git a/fern/docs/pages/airdrop/local-development.mdx b/fern/docs/pages/airdrop/local-development.mdx index 7f664e2b..77b04257 100644 --- a/fern/docs/pages/airdrop/local-development.mdx +++ b/fern/docs/pages/airdrop/local-development.mdx @@ -1,10 +1,9 @@ -For easier development you can run your snap-in locally and receive logs directly in your console. +For easier development you can run your Airdrop snap-in locally and receive logs directly in your console. ## Prerequisites - [devrev-cli](https://developer.devrev.ai/snapin-development/references/cli-install) (version 4.7 or higher) - [jq](https://stedolan.github.io/jq) -- [golang](https://go.dev/doc/install) - [nodejs](https://nodejs.org/en/download/package-manager) (version 18.x.x+ or higher) - [ngrok](https://ngrok.com) @@ -50,7 +49,7 @@ DevRev provides a starter template, which you can run and test out right away. Now that you have a running snap-in, you can start an airdrop. Go to DevRev app and click **Airdrops** -> **Start Airdrop** -> **Your snap-in**. -1. Starting an airdrop begins with establishing a connection to the external system. The snap-in prompts for specific information needed to create this connection. Afterwards, the snap-in verifies that the connection is successful and you should see a success message. +1. Starting an airdrop begins with establishing a connection to the external system. The Airdrop snap-in prompts for specific information needed to create this connection. Afterwards, the snap-in verifies that the connection is successful and you should see a success message. 2. Once the connection is established, the snap-in collects information from the external system about the data unit to be exported. This phase is called _external sync units extraction_. An external sync unit is the data unit from the external system to be imported to DevRev. At this point, the DevRev app prompts the user to select an external sync unit from the list. @@ -64,4 +63,4 @@ Go to DevRev app and click **Airdrops** -> **Start Airdrop** -> **Your snap-in** The process described above is called the **initial sync**. In all subsequent syncs, the first two steps are skipped. -The snap-in constantly communicates with Airdrop, which manages its behavior. When an action occurs in the UI, Airdrop sends a message to the snap-in with details about the event. The snap-in performs an action based on the event type. For example, when a user starts a new airdrop for the first time, Airdrop instructs the snap-in to perform an external sync unit extraction. The snap-in carries out the task and sends back available sync units to Airdrop. +The snap-in constantly communicates with Airdrop platform, which manages its behavior. When an action occurs in the DevRev app, Airdrop platform sends a message to the snap-in with details about the event. The snap-in performs an action based on the event type. For example, when a user starts a new airdrop for the first time, Airdrop platform instructs the snap-in to perform an external sync unit extraction. The snap-in carries out the task and sends back available sync units to Airdrop platform. diff --git a/fern/docs/pages/airdrop/manifest.mdx b/fern/docs/pages/airdrop/manifest.mdx index e3e3482a..c8e1b5be 100644 --- a/fern/docs/pages/airdrop/manifest.mdx +++ b/fern/docs/pages/airdrop/manifest.mdx @@ -3,8 +3,8 @@ Start developing your snap-in with a `manifest.yaml` file. The manifest defines Begin by setting the values for `name` and `description` in the template. ```yaml -name: ADaaS Template -description: A template for creating a new ADaaS snap-in +name: Template Connector +description: Template Connector for importing dummy data into DevRev ``` ### Set snap-in functionalities @@ -29,9 +29,9 @@ Check the `imports` section next. ```yaml imports: - - slug: adaas-template - display_name: ADaaS Connector Template - description: ADaaS Connector Template + - slug: airdrop-template-snap-in + display_name: Template Connector + description: Template Connector for importing data into DevRev extractor_function: extraction loader_function: loading allowed_connection_types: diff --git a/fern/docs/pages/airdrop/metadata-extraction.mdx b/fern/docs/pages/airdrop/metadata-extraction.mdx index cbdfd22d..16c25eef 100644 --- a/fern/docs/pages/airdrop/metadata-extraction.mdx +++ b/fern/docs/pages/airdrop/metadata-extraction.mdx @@ -3,6 +3,8 @@ During the metadata extraction phase, the snap-in must provide an This file provides a structured way of describing the external system's domain model, including its domain entities, types, relationships, and other metadata. +You can check which object types for Airdrop DevRev supports [here](/snapin-development/airdrop/supported-object-types). + ## Triggering event Airdrop initiates the metadata extraction by starting the snap-in with a message with an event diff --git a/fern/docs/pages/airdrop/overview.mdx b/fern/docs/pages/airdrop/overview.mdx index dd554e7d..fdf1b7b7 100644 --- a/fern/docs/pages/airdrop/overview.mdx +++ b/fern/docs/pages/airdrop/overview.mdx @@ -4,7 +4,8 @@ between DevRev and the external systems. DevRev provides a range of snap-ins available through the DevRev marketplace for integration with various external systems like Jira and Zendesk. Sometimes, a specific snap-in may not be available -for a given external system. This is why DevRev enables creation of external workers (extractors -and loaders) to other developers, allowing them to integrate with DevRev’s Airdrop and develop their own snap-ins. +for a given external system. This is why DevRev enables creation of external connectors (extractors +and loaders) to other developers, allowing them to integrate with DevRev’s Airdrop and develop their +Airdrop own snap-ins. You can learn more about Airdrop in the [general documentation](https://docs.devrev.ai/import#airdrop-features). diff --git a/fern/docs/pages/airdrop/publish-to-marketplace.mdx b/fern/docs/pages/airdrop/publish-to-marketplace.mdx index d836e0df..b98d3156 100644 --- a/fern/docs/pages/airdrop/publish-to-marketplace.mdx +++ b/fern/docs/pages/airdrop/publish-to-marketplace.mdx @@ -10,19 +10,19 @@ devrev marketplace_submissions create ### 2. Submit marketplace listing for review -The newly created marketplace submission is in the Draft state. In order to publish it, it first has to transition to Waiting for review state. This can be done by running the following command, which starts a wizard to guide you through the process: +The newly created marketplace submission is in the _Draft_ state. In order to publish it, it first has to transition to _Waiting for review_ state. This can be done by running the following command, which starts a wizard to guide you through the process: ```bash devrev marketplace_submissions transition ``` -Once the submission is transitioned to the Waiting for review state, it needs to be approved by a marketplace admin. While in review, the state of the submission is In review. Once it is reviewed, the state changes to either Approved or Rejected. +Once the submission is transitioned to the _Waiting for review_ state, it needs to be approved by a marketplace admin. While in review, the state of the submission is _In review_. Once it is reviewed, the state changes to either _Approved_ or _Rejected_. Submissions whose state is _In review_ or _Approved_ cannot be modified. -If the submission was rejected, you can transition it back to Draft and modify it to satisfy the requirements. Once it is updated, you can transition it back to Waiting for review to be reviewed again. +If the submission was rejected, you can transition it back to _Draft_ and modify it to satisfy the requirements. Once it is updated, you can transition it back to _Waiting for review_ to be reviewed again. ### 3. Publish marketplace submission diff --git a/fern/docs/pages/airdrop/snap-in-template.mdx b/fern/docs/pages/airdrop/snap-in-template.mdx index 5af322eb..eaa222ca 100644 --- a/fern/docs/pages/airdrop/snap-in-template.mdx +++ b/fern/docs/pages/airdrop/snap-in-template.mdx @@ -1,17 +1,18 @@ -For easier snap-in development, a starter template is available [here](https://github.com/devrev/adaas-template). +Airdrop snap-in starter template is available [here](https://github.com/devrev/adaas-template). -The template makes use of **DevRev's SDK library**, which helps developers build snap-ins that -integrate with DevRev’s Airdrop platform. This library simplifies the workflow for handling -data extraction and loading, event-driven actions, state management, and artifact handling. +The template makes use of **[DevRev's SDK library](https://www.npmjs.com/package/@devrev/ts-adaas)**, +which helps developers build snap-ins that integrate with DevRev’s Airdrop platform. +This library simplifies the workflow for handling data extraction and loading, event-driven actions, +state management, and artifact handling. It provides features such as: -- **Type definitions**: Structured types for Airdrop control protocol. -- **Event management**: Communication between Airdrop and the snap-in involves messages called +- **Event management**: Communication between Airdrop platform and Airdrop snap-in involves messages called _events_ - depending on the type of event, the snap-in performs different actions. - **State handling**: Update and access state in real-time within tasks. - **Artifact management**: Supports batched storage of artifacts. - **Error & timeout support**: Error handling and timeout management for long-running tasks. +- **Type definitions**: Structured types for Airdrop control protocol. ### Snap-in code structure @@ -45,5 +46,3 @@ If you check the template, you can see that a snap-in is a Node.js project with The `manifest.yaml` is a configuration file that defines details such as the snap-in's name, description, version, connection information, and functionality specifics. Adjusting this file is the initial step in the development of a snap-in. The `code/` folder holds the implementation of the snap-in. Inside, the `functions/` subfolder includes the logic for retrieving external sync units, extracting data from and loading data to external systems, and mapping between them. - -The `scripts/` folder contains the scripts necessary for the building and deployment of the snap-in, streamlining the development and deployment processes with the help of tools like `deploy.sh` and `build.sh`. diff --git a/fern/docs/pages/airdrop/supported-object-types.mdx b/fern/docs/pages/airdrop/supported-object-types.mdx new file mode 100644 index 00000000..48e0acc0 --- /dev/null +++ b/fern/docs/pages/airdrop/supported-object-types.mdx @@ -0,0 +1,1147 @@ +# Supported DevRev object types for Airdrop + +Last updated: _2025-04_ + +This document contains the list of all DevRev object types that are supported for Airdrop. + +## Summary + +| Object Type | Object Hierarchy | Loading | Extraction | Subtype | AppFragment | Comment | +|-------|-------------|:-------:|:----------:|:-------:|:----------:|-------------| +| [account](#account) | account | ✔︎ | | | | | +| [airdrop_authorization_policy](#airdrop_authorization_policy) | airdrop_authorization_policy | ✔︎ | | | | _Airdrop authorization policy object type is the intermediate object type for creating MFZ object types: role, role_set, object_member, and access_control_entry._ | +| [airdrop_platform_group](#airdrop_platform_group) | airdrop_platform_group | ✔︎ | | | | _Airdrop platform group object type is the representation of the default DevRev groups._ | +| [article](#article) | article | ✔︎ | | | | | +| [capability](#capability) | part->capability | ✔︎ | ✔︎ | ✔︎ | | | +| [channel](#channel) | chat->channel | ✔︎ | | | | | +| [comment](#comment) | comment | ✔︎ | ✔︎ | | | | +| [component](#component) | part->component | ✔︎ | ✔︎ | ✔︎ | | | +| [conversation](#conversation) | conversation | ✔︎ | | ✔︎ | | | +| [custom_link](#custom_link) | link | ✔︎ | ✔︎ | | | | +| [custom_part](#custom_part) | part->custom_part | ✔︎ | ✔︎ | ✔︎ | | | +| [devu](#devu) | dev_user | ✔︎ | | | | | +| [directory](#directory) | directory | ✔︎ | | | | | +| [dm](#dm) | chat->dm | ✔︎ | | | | | +| [enhancement](#enhancement) | part->enhancement | ✔︎ | ✔︎ | ✔︎ | | | +| [feature](#feature) | part->feature | ✔︎ | ✔︎ | ✔︎ | | | +| [group](#group) | group | ✔︎ | | | | | +| [incident](#incident) | incident | ✔︎ | ✔︎ | ✔︎ | | | +| [issue](#issue) | work->issue | ✔︎ | ✔︎ | ✔︎ | | | +| [link](#link) | link | ✔︎ | ✔︎ | | | | +| [linkable](#linkable) | part->linkable | ✔︎ | ✔︎ | ✔︎ | | | +| [meeting](#meeting) | meeting | ✔︎ | | | | | +| [microservice](#microservice) | part->microservice | ✔︎ | ✔︎ | ✔︎ | | | +| [object_member](#object_member) | object_member | ✔︎ | | | | | +| [opportunity](#opportunity) | work->opportunity | ✔︎ | ✔︎ | ✔︎ | | | +| [product](#product) | part->product | ✔︎ | ✔︎ | ✔︎ | | | +| [revu](#revu) | rev_user | ✔︎ | | | | | +| [runnable](#runnable) | part->runnable | ✔︎ | ✔︎ | ✔︎ | | | +| [sysu](#sysu) | sys_user | ✔︎ | | | | | +| [tag](#tag) | tag | ✔︎ | | | | | +| [task](#task) | work->task | ✔︎ | ✔︎ | ✔︎ | | | +| [test_custom_type](#test_custom_type) | custom_object | ✔︎ | | ✔︎ | | _Test custom object type for testing purposes. Available only for old mappings._ | +| [ticket](#ticket) | work->ticket | ✔︎ | ✔︎ | ✔︎ | | | + +## Key Capabilities + +| Capability | Description | +|-------|-------------| +| Can Load | Indicates that the object type can be loaded into the DevRev with Airdrop. | +| Can Extract | Indicates that the object type can be synced out to external systems. | +| Can Subtype | Indicates that the object type supports [subtype](https://developer.devrev.ai/beta/guides/object-customization#subtype) creation. | +| Can App Fragment | Indicates that the object type supports application fragments. | + +## account + +**Resource Type:** `account` + +**Capabilities:** Can Load + +### Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `address` | struct | | Address of the Organization. | +| `created_by_id` | reference -> [#category:user] | | User ID of the user that created the object | +| `description` | rich_text | | Description of the corresponding Account. | +| `display_name` | text | ✔︎ | Name of the Organization. | +| `environment` | enum | | The environment of the Org. Defaults to 'production' if not specified. | +| `owned_by` | reference (collection) -> [#category:user] | ✔︎ | List of Dev user IDs owning this Account. | +| `phone_number` | struct (collection) | | Phone numbers of the Organization. | +| `phone_numbers` | text (collection) | | Phone numbers of the Organization. | +| `state` | enum | ✔︎ | State of the Organization. | +| `tags` | reference (collection) -> [#record:tag] | | Tags associated with an object. | +| `websites` | text (collection) | | Websites | + +### Structs + +#### postal_address + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `country` | text | | Country name component. | +| `formatted` | text | | Full mailing address, formatted for display or use on a mailing label. | +| `locality` | text | | Town, city. | +| `postal_code` | text | | Zip code of the address. | +| `region` | text | | State, province, prefecture, or region component. | +| `street_address` | text | | Full street address component. | + + +[ ▲ top](#summary) + +--- + +## airdrop_authorization_policy + +**Resource Type:** `airdrop_authorization_policy` + +**Capabilities:** Can Load + +### Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `groups` | reference (collection) -> [#record:airdrop_platform_group #record:group] | | Groups this role applies to. | +| `privileges` | enum (collection) | | Privileges the role provides. | +| `targets` | type_key (collection) | ✔︎ | Target object types the role applies to. | +| `users` | reference (collection) -> [#record:devu #record:revu] | | Users this role applies to. | + + +[ ▲ top](#summary) + +--- + +## airdrop_platform_group + +**Resource Type:** `airdrop_platform_group` + +**Capabilities:** Can Load + +### Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `key` | enum | ✔︎ | Key of the platform provided group. | + + +[ ▲ top](#summary) + +--- + +## article + +**Resource Type:** `article` + +**Capabilities:** Can Load + +### Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `access_level` | enum | | Default access level for the object. | +| `aliases` | text (collection) | | Aliases for the URL of the article. | +| `applies_to_part_ids` | reference (collection) -> [#category:part] | | Parts relevant to the article. | +| `authored_by_ids` | reference (collection) -> [#category:user] | | Users that authored the article. | +| `content` | rich_text | | | +| `content_filename` | text | | | +| `content_filetype` | enum | | | +| `created_by_id` | reference -> [#category:user] | | User ID of the user that created the object | +| `description` | rich_text | | Description of the article. | +| `extracted_content` | reference (collection) -> [#record:artifact] | | Extracted content of the article. | +| `extracted_plain_text` | text | | Extracted Plain Text | +| `item_url_field` | text | | Link to the item in the origin system | +| `language` | text | | Language of the article for i18n support. | +| `modified_by_id` | reference -> [#category:user] | | User ID of the user that last modified the object | +| `num_downvotes` | int | | Number of downvotes on the article. | +| `num_upvotes` | int | | Number of upvotes on the article. | +| `owned_by_ids` | reference (collection) -> [#category:user] | ✔︎ | Users that own the article. | +| `parent` | reference -> [#record:directory] | | Parent of the article. | +| `primary_article_id` | reference -> [#record:article] | | Primary article ID | +| `published_date` | timestamp | | Timestamp when the article was published. | +| `rank` | text | | Rank of the article. | +| `resource` | struct | | Resource of the article. | +| `scope` | enum | | Scope of the article. | +| `shared_with` | permission (collection) | | The list of Rev user, groups and dynamic groups with whom the article is shared and the corresponding roles. | +| `status` | enum | | Status of the article. | +| `tags` | reference (collection) -> [#record:tag] | | Tags associated with the article. | +| `title` | text | ✔︎ | Title of the article. | +| `type` | enum | | Type of the article. | +| `visible_to` | enum | | | + +### Structs + +#### resource + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `artifact_ids` | reference | | Artifacts of the resource (relevant only for type artifact). | +| `artifacts` | reference | | Artifacts of the resource (relevant only for type artifact). | +| `content_blocks` | reference | | Content blocks used in the article (relevant only for devrev/rt articles). | +| `published_version` | text | | The published version of the resource | +| `type` | enum | | The type of resource. | +| `url` | text | | URL of the resource (relevant only for type url). | + + +[ ▲ top](#summary) + +--- + +## capability + +**Resource Type:** `part`, `capability` + +**Capabilities:** Can Extract, Can Load, Can Subtype + +### Fixed Fields + +| Field | Value | +|-------|-------| +| `part_type` | "capability" | + +### Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `created_by_id` | reference -> [#category:user] | | User ID of the user that created the object | +| `delivered_as` | enum (collection) | | Methods the product can be delivered as | +| `description` | rich_text | | Description of the part | +| `docs` | struct (collection) | | Docs associated with the part | +| `fulfilled_by` | reference (collection) -> [#category:part] | | IDs of the runnables that fulfill this capability | +| `item_url_field` | text | | Link to the item in the origin system | +| `modified_by_id` | reference -> [#category:user] | | User ID of the user that last modified the object | +| `name` | text | ✔︎ | Name of the part | +| `owned_by_ids` | reference (collection) -> [#record:devu] | ✔︎ | User IDs of the users that own the part | +| `parent_part` | reference -> [#category:part] | ✔︎ | Parent of this part | +| `part_of_part_id` | reference -> [#category:part] | | Product ID of this Capability | +| `pm_owner_id` | reference -> [#record:devu] | | User ID of the PM owner of the part | +| `ref_url` | text | | URL to the part details (git url, website, etc.) | +| `tags` | reference (collection) -> [#record:tag] | | Tags associated with the object | + + +[ ▲ top](#summary) + +--- + +## channel + +**Resource Type:** `chat`, `channel` + +**Capabilities:** Can Load + +### Fixed Fields + +| Field | Value | +|-------|-------| +| `chat_type` | "channel" | + +### Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `access_level` | enum | | The access level for the channel. | +| `created_by_id` | reference -> [#category:user] | | User ID of the user that created the object | +| `name` | text | | The name of the channel. If provided, this will be unique across all channels in the Dev org. It's case insensitive and must contain the characters [A-Za-z0-9_-]. | +| `title` | text | | The title given to the chat. | + + +[ ▲ top](#summary) + +--- + +## comment + +**Resource Type:** `comment` + +**Capabilities:** Can Extract, Can Load + +### Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `body` | rich_text | ✔︎ | Comment body | +| `created_by_id` | reference -> [#category:user] | ✔︎ | User ID of the user that created the object | +| `creator_display_name` | text | | Creator display name | +| `external_ref` | text | | The external reference to associate with the entry. This must be unique across the entry's siblings, i.e. all children of its parent. | +| `grandparent_object_id` | reference -> [#category:chat #category:part #category:work #record:account #record:conversation #record:incident #record:revu] | | The object this relates to | +| `grandparent_object_type` | text | | The object type this relates to | +| `item_url_field` | text | | Link to the item in the origin system | +| `modified_by_id` | reference -> [#category:user] | ✔︎ | User ID of the user that last modified the object | +| `parent_object_id` | reference -> [#category:chat #category:part #category:work #record:account #record:comment #record:conversation #record:incident #record:revu] | ✔︎ | The object this relates to | +| `parent_object_type` | text | | The object type this relates to | +| `visibility` | enum | | Visibility of the comment | + + +[ ▲ top](#summary) + +--- + +## component + +**Resource Type:** `part`, `component` + +**Capabilities:** Can Extract, Can Load, Can Subtype + +### Fixed Fields + +| Field | Value | +|-------|-------| +| `part_type` | "component" | + +### Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `created_by_id` | reference -> [#category:user] | | User ID of the user that created the object | +| `delivered_as` | enum (collection) | | Methods the product can be delivered as | +| `description` | rich_text | | Description of the part | +| `development_owner_id` | reference -> [#record:devu] | | User ID of the development owner of the part | +| `docs` | struct (collection) | | Docs associated with the part | +| `item_url_field` | text | | Link to the item in the origin system | +| `modified_by_id` | reference -> [#category:user] | | User ID of the user that last modified the object | +| `name` | text | ✔︎ | Name of the part | +| `owned_by_ids` | reference (collection) -> [#record:devu] | ✔︎ | User IDs of the users that own the part | +| `parent_part` | reference -> [#category:part] | | Parent of this part | +| `pm_owner_id` | reference -> [#record:devu] | | User ID of the PM owner of the part | +| `qa_owner_id` | reference -> [#record:devu] | | User ID of the QA owner of the part | +| `ref_url` | text | | URL to the part details (git url, website, etc.) | +| `tags` | reference (collection) -> [#record:tag] | | Tags associated with the object | + + +[ ▲ top](#summary) + +--- + +## conversation + +**Resource Type:** `conversation` + +**Capabilities:** Can Load, Can Subtype + +### Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `applies_to_part_ids` | reference (collection) -> [#category:part] | | Details of the parts relevant to the conversation. | +| `broadcast_channels` | text (collection) | | Active channels for the conversation | +| `channels` | reference (collection) -> [#record:external_communication_channel] | | Channel IDs of the conversation | +| `conversation_type` | enum | ✔︎ | A type of object used to track conversations | +| `created_by_id` | reference -> [#category:user] | | User ID of the user that created the object | +| `description` | rich_text | | Description of the conversation object | +| `first_response_time` | timestamp | | Timestamp to track the first response from Dev | +| `first_unreplied_rev_message_date` | timestamp | | First unreplied customer message's timestamp | +| `group` | reference -> [#record:group] | | Group that owns the conversation | +| `is_creator_verified` | bool | | Whether the conversation is created by verified user. | +| `is_spam` | bool | | Whether the conversation is spam. | +| `last_external_message_date` | timestamp | | Timestamp of the last message in external discussion | +| `last_external_message_id` | reference -> [#record:comment] | | ID of the last message in external discussion. | +| `last_internal_message_date` | timestamp | | Timestamp of the last message in internal discussions. | +| `last_internal_message_id` | reference -> [#record:comment] | | ID of the last message in internal discussion. | +| `last_message_id` | reference -> [#record:comment] | | ID of the last message on the object. | +| `last_message_timestamp` | timestamp | | Timestamp of the last message in the conversation | +| `member_ids` | reference (collection) -> [#category:user] | ✔︎ | User IDs of the users in the conversation | +| `modified_by_id` | reference -> [#category:user] | | User ID of the user that last modified the object | +| `owned_by_ids` | reference (collection) -> [#category:user] | | Users that own the conversation | +| `participant_oids` | reference (collection) -> [#record:account #record:revo] | | Globally unique IDs of participating orgs | +| `participant_uids` | reference (collection) -> [#category:user] | | Globally unique ID of DevRev rev users | +| `priority` | enum | | Priority of the conversation | +| `sla_id` | reference -> [#record:sla] | | The ID of the SLA applying to this object, controlling how the SLA tracking service treats it. | +| `sla_tracker_id` | reference -> [#record:sla_tracker] | | The ID of the SLA tracker applied to this object, which is used to track the SLA metrics. | +| `source_channel` | text | | Source channel for the conversation | +| `source_channel_v2` | reference -> [#record:external_communication_channel] | | Source channel ID of the conversation | +| `stage` | enum | | Stage of the conversation | +| `started_by_id` | reference -> [#category:user] | | User that started the conversation | +| `tags` | reference (collection) -> [#record:tag] | | Tags associated with the object | +| `title` | text | | Title of the conversation object | +| `turing_active` | bool | | Whether Turing is active on the conversation. Will be true when Turing is active and false when it stops responding or when a dev user responds to the conversation. | +| `user_sessions` | reference (collection) -> [#record:user_session] | | User sessions associated with the conversation. | + + +[ ▲ top](#summary) + +--- + +## custom_link + +**Resource Type:** `link` + +**Capabilities:** Can Extract, Can Load + +### Fixed Fields + +| Field | Value | +|-------|-------| +| `link_type` | "custom_link" | + +### Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `created_by_id` | reference -> [#category:user] | | User ID of the user that created the object | +| `link_type_id` | enum | ✔︎ | Link Type | +| `source_id` | reference -> [#record:account #record:capability #record:enhancement #record:feature #record:incident #record:issue #record:opportunity #record:product #record:project #record:revu #record:task #record:ticket] | ✔︎ | Source object ID | +| `target_id` | reference -> [#record:account #record:capability #record:enhancement #record:feature #record:incident #record:issue #record:opportunity #record:product #record:project #record:revu #record:task #record:ticket] | ✔︎ | Target object ID | + + +[ ▲ top](#summary) + +--- + +## custom_part + +**Resource Type:** `part`, `custom_part` + +**Capabilities:** Can Extract, Can Load, Can Subtype + +### Fixed Fields + +| Field | Value | +|-------|-------| +| `part_type` | "custom_part" | + +### Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `created_by_id` | reference -> [#category:user] | | User ID of the user that created the object | +| `delivered_as` | enum (collection) | | Methods the product can be delivered as | +| `description` | rich_text | | Description of the part | +| `docs` | struct (collection) | | Docs associated with the part | +| `item_url_field` | text | | Link to the item in the origin system | +| `modified_by_id` | reference -> [#category:user] | | User ID of the user that last modified the object | +| `name` | text | ✔︎ | Name of the part | +| `owned_by_ids` | reference (collection) -> [#record:devu] | ✔︎ | User IDs of the users that own the part | +| `parent_part` | reference -> [#category:part] | | Parent of this part | +| `ref_url` | text | | URL to the part details (git url, website, etc.) | +| `tags` | reference (collection) -> [#record:tag] | | Tags associated with the object | + + +[ ▲ top](#summary) + +--- + +## devu + +**Resource Type:** `dev_user` + +**Capabilities:** Can Load + +### Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `display_name` | text | ✔︎ | The user's display name. The name is non-unique and mutable. | +| `email` | text | | Email address of the user. | +| `external_identities` | struct (collection) | | IDs of the Dev User outside the DevRev SOR. | +| `full_name` | text | | Full name of the user. | +| `state` | enum | | State of the user. | + + +[ ▲ top](#summary) + +--- + +## directory + +**Resource Type:** `directory` + +**Capabilities:** Can Load + +### Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `body` | reference -> [#record:artifact] | | Body of the directory | +| `created_by_id` | reference -> [#category:user] | | User ID of the user that created the object | +| `description` | rich_text | | Description of the directory | +| `icon` | text | | Icon of the directory | +| `language` | text | | Language of the directory | +| `modified_by_id` | reference -> [#category:user] | | User ID of the user that last modified the object | +| `parent` | reference -> [#record:directory] | | Parent directory | +| `published` | bool | ✔︎ | Whether the directory is published | +| `rank` | text | | Rank of the directory | +| `tags` | reference (collection) -> [#record:tag] | | Tags associated with the directory | +| `thumbnail` | reference -> [#record:artifact] | | Thumbnail of the directory | +| `title` | text | ✔︎ | Title of the directory | + + +[ ▲ top](#summary) + +--- + +## dm + +**Resource Type:** `chat`, `dm` + +**Capabilities:** Can Load + +### Fixed Fields + +| Field | Value | +|-------|-------| +| `chat_type` | "dm" | + +### Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `created_by_id` | reference -> [#category:user] | | User ID of the user that created the object | +| `is_default` | bool | | Whether this is the default DM for messaging the constituent users. If true, then this DM is always returned when opening a DM for the users. Note only one DM may be the default for a given set of users. | +| `record_ids` | text (collection) | | The associated records for this DM. | +| `title` | text | | The title given to the chat. | +| `user_ids` | reference (collection) -> [#category:user] | ✔︎ | The users in the direct message. Only these users have access the chat's history and can send new messages. | + + +[ ▲ top](#summary) + +--- + +## enhancement + +**Resource Type:** `part`, `enhancement` + +**Capabilities:** Can Extract, Can Load, Can Subtype + +### Fixed Fields + +| Field | Value | +|-------|-------| +| `part_type` | "enhancement" | + +### Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `account_ids` | reference (collection) -> [#record:account] | | Accounts associated to enhancement | +| `created_by_id` | reference -> [#category:user] | | User ID of the user that created the object | +| `delivered_as` | enum (collection) | | Methods the product can be delivered as | +| `description` | rich_text | | Description of the part | +| `docs` | struct (collection) | | Docs associated with the part | +| `item_url_field` | text | | Link to the item in the origin system | +| `modified_by_id` | reference -> [#category:user] | | User ID of the user that last modified the object | +| `name` | text | ✔︎ | Name of the part | +| `opportunity_ids` | reference (collection) -> [#category:work] | | Opportunities from the enhancement | +| `owned_by_ids` | reference (collection) -> [#record:devu] | ✔︎ | User IDs of the users that own the part | +| `parent_part` | reference -> [#category:part] | ✔︎ | Parent of this part | +| `potential_revenue` | float | | Potential revenue from the enhancement | +| `ref_url` | text | | URL to the part details (git url, website, etc.) | +| `release_notes` | text | | Release notes of the enhancement | +| `rev_score` | float | | Rev Score of the enhancement | +| `rev_score_tier` | enum | | Rev Score tier of the enhancement | +| `stage` | enum | | Stage of the part | +| `tags` | reference (collection) -> [#record:tag] | | Tags associated with the object | +| `target_close_date` | timestamp | | Target close date for the object | +| `target_start_date` | timestamp | | Target start date for the object | +| `ticket_ids` | reference (collection) -> [#category:work] | | Tickets associated with the enhancement | + + +[ ▲ top](#summary) + +--- + +## feature + +**Resource Type:** `part`, `feature` + +**Capabilities:** Can Extract, Can Load, Can Subtype + +### Fixed Fields + +| Field | Value | +|-------|-------| +| `part_type` | "feature" | + +### Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `api_endpoints` | struct (collection) | | Associated API endpoints | +| `api_parameters_summary` | text (collection) | | Associated API parameters | +| `api_prefix_summary` | text (collection) | | Common Path Denominators of the Api Operations | +| `created_by_id` | reference -> [#category:user] | | User ID of the user that created the object | +| `delivered_as` | enum (collection) | | Methods the product can be delivered as | +| `description` | rich_text | | Description of the part | +| `development_owner_id` | reference -> [#record:devu] | | User ID of the development owner of the part | +| `discoveryevidences` | text (collection) | | A evidences that the inferer were able to find that justify the inference outcome | +| `docs` | struct (collection) | | Docs associated with the part | +| `fulfilled_by` | reference (collection) -> [#category:part] | | IDs of the runnables that fulfill this feature | +| `item_url_field` | text | | Link to the item in the origin system | +| `modified_by_id` | reference -> [#category:user] | | User ID of the user that last modified the object | +| `name` | text | ✔︎ | Name of the part | +| `owned_by_ids` | reference (collection) -> [#record:devu] | ✔︎ | User IDs of the users that own the part | +| `parent_part` | reference -> [#category:part] | ✔︎ | Parent of this part | +| `part_of_part_id` | reference -> [#category:part] | | Capability ID or Parent Feature ID of this feature | +| `pm_owner_id` | reference -> [#record:devu] | | User ID of the PM owner of the part | +| `qa_owner_id` | reference -> [#record:devu] | | User ID of the QA owner of the part | +| `ref_url` | text | | URL to the part details (git url, website, etc.) | +| `tags` | reference (collection) -> [#record:tag] | | Tags associated with the object | + + +[ ▲ top](#summary) + +--- + +## group + +**Resource Type:** `group` + +**Capabilities:** Can Load + +### Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `created_by_id` | reference -> [#category:user] | | User ID of the user that created the object | +| `description` | rich_text | ✔︎ | Description of the group | +| `includes` | reference (collection) -> [#record:group] | | IDs of the group(s) that the group includes | +| `member_type` | enum | | Type of the members in the group. | +| `name` | text | ✔︎ | Name of the group | +| `owner` | reference -> [#record:devu] | | Owner of the group | +| `type` | enum | | Type of the group | + + +[ ▲ top](#summary) + +--- + +## incident + +**Resource Type:** `incident` + +**Capabilities:** Can Extract, Can Load, Can Subtype + +### Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `acknowledged_date` | timestamp | | Timestamp when the incident was acknowledged. | +| `applies_to_part_ids` | reference (collection) -> [#category:part] | | Parts to which the work is attached. | +| `body` | rich_text | | Body of the work object | +| `created_by_id` | reference -> [#category:user] | | User ID of the user that created the object | +| `identified_date` | timestamp | | Time when the incident was identified/reported. | +| `item_url_field` | text | | Link to the item in the origin system | +| `mitigated_date` | timestamp | | Timestamp when the incident was mitigated. | +| `owned_by_ids` | reference -> [#category:user] | ✔︎ | User IDs of the users that own the work | +| `pia_ids` | reference (collection) -> [#record:article] | | The article ids of the Post-Incident Analysis(PIA) of the incident. | +| `playbook_ids` | reference (collection) -> [#record:article] | | The article ids of the playbook(s) associated with the incident. | +| `related_doc_ids` | reference (collection) -> [#record:article] | | The article ids of other documents associated with the incident. | +| `reported_by` | enum | | The entity that first reported the incident. | +| `severity` | enum | ✔︎ | Severity of the work | +| `stage` | enum | ✔︎ | | +| `tags` | reference (collection) -> [#record:tag] | | Tags associated with the object. | +| `target_close_date` | timestamp | | Timestamp when the work is expected to be complete | +| `title` | text | ✔︎ | Title of the work object | + + +[ ▲ top](#summary) + +--- + +## issue + +**Resource Type:** `work`, `issue` + +**Capabilities:** Can Extract, Can Load, Can Subtype + +### Fixed Fields + +| Field | Value | +|-------|-------| +| `work_type` | "issue" | + +### Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `account_ids` | reference (collection) -> [#record:account] | | Accounts associated to issues | +| `actual_close_date` | timestamp | | Timestamp when the work was actually completed | +| `actual_effort` | float | | Actual effort to complete the issue | +| `applies_to_part_id` | reference -> [#category:part] | ✔︎ | Details of the part relevant to the work | +| `applies_to_version_ids` | reference (collection) -> [#record:version] | | Part versions relevant to the work | +| `body` | rich_text | | Body of the work object | +| `created_by_id` | reference -> [#category:user] | | User ID of the user that created the object | +| `estimated_effort` | float | | Estimated effort to complete the issue | +| `fallback_parts` | reference -> [#category:part] | | Other values that could serve as part if apples_to_part_id is found to be invalid. | +| `item_url_field` | text | | Link to the item in the origin system | +| `modified_by_id` | reference -> [#category:user] | | User ID of the user that last modified the object | +| `owned_by_ids` | reference (collection) -> [#category:user] | ✔︎ | User IDs of the users that own the work | +| `priority` | enum | ✔︎ | Priority of the work based upon impact and criticality | +| `release_version_ids` | reference (collection) -> [#record:version] | | Versions that will contain the resolving commit IDs | +| `reported_by_ids` | reference (collection) -> [#category:user] | | User IDs of the users that reported the work | +| `rev_org_ids` | reference (collection) -> [#record:account #record:revo] | | Rev orgs associated to issues | +| `sla_id` | reference -> [#record:sla] | | The ID of the SLA applying to this object, controlling how the SLA tracking service treats it. | +| `sla_tracker_id` | reference -> [#record:sla_tracker] | | The ID of the SLA tracker applied to this object, which is used to track the SLA metrics. | +| `sprint_id` | reference -> [#record:vista_group_item] | | Sprint to which the issue belongs. | +| `stage` | enum | ✔︎ | Stage of the object | +| `tags` | reference (collection) -> [#record:tag] | | Tags associated with the object | +| `target_close_date` | timestamp | | Timestamp when the work is expected to be complete | +| `target_start_date` | timestamp | | Target start date for the object | +| `title` | text | ✔︎ | Title of the work object | + + +[ ▲ top](#summary) + +--- + +## link + +**Resource Type:** `link` + +**Capabilities:** Can Extract, Can Load + +### Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `created_by_id` | reference -> [#category:user] | | User ID of the user that created the object | +| `link_type` | enum | ✔︎ | Type of link used to define the relationship. | +| `source_id` | reference -> [#category:work #record:incident] | ✔︎ | Source object ID | +| `source_object_type` | enum | | Source object type | +| `target_id` | reference -> [#category:work #record:incident] | ✔︎ | Target object ID | +| `target_object_type` | enum | | Target object type | +| `transformed_from_id` | text | | | + + +[ ▲ top](#summary) + +--- + +## linkable + +**Resource Type:** `part`, `linkable` + +**Capabilities:** Can Extract, Can Load, Can Subtype + +### Fixed Fields + +| Field | Value | +|-------|-------| +| `part_type` | "linkable" | + +### Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `coderepo_paths` | text | | Paths in the repository for the code part | +| `coderepo_url` | text | | URL to the server & repo for the code part | +| `created_by_id` | reference -> [#category:user] | | User ID of the user that created the object | +| `delivered_as` | enum (collection) | | Methods the product can be delivered as | +| `description` | rich_text | | Description of the part | +| `discoveryevidences` | text (collection) | | A evidences that the inferer were able to find that justify the inference outcome | +| `docs` | struct (collection) | | Docs associated with the part | +| `item_url_field` | text | | Link to the item in the origin system | +| `kind` | enum | | The kind of linkable | +| `modified_by_id` | reference -> [#category:user] | | User ID of the user that last modified the object | +| `name` | text | ✔︎ | Name of the part | +| `owned_by_ids` | reference (collection) -> [#record:devu] | ✔︎ | User IDs of the users that own the part | +| `parent_part` | reference -> [#category:part] | | Parent of this part | +| `ref_url` | text | | URL to the part details (git url, website, etc.) | +| `tags` | reference (collection) -> [#record:tag] | | Tags associated with the object | + + +[ ▲ top](#summary) + +--- + +## meeting + +**Resource Type:** `meeting` + +**Capabilities:** Can Load + +### Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `channel` | enum | ✔︎ | The channel of meeting. | +| `created_by_id` | reference -> [#category:user] | | User ID of the user that created the object | +| `description` | rich_text | | Description of the meeting. | +| `ended_date` | timestamp | | Time at which meeting ended. | +| `engagement_new_ref` | text | | Reference ID associated with the new engagement | +| `external_ref_id` | text | | External reference of the meeting. This is the identifier from the meeting channel/provider. | +| `external_url` | text | | External URL associated with the meeting. | +| `member_ids` | reference (collection) -> [#category:user] | ✔︎ | IDs of the members in the meeting. | +| `organizer_id` | reference -> [#category:user] | ✔︎ | ID of the organizer of the meeting. | +| `parent_id` | reference -> [#category:work #record:account] | | Parent ID of the meeting. | +| `parent_object_type` | enum | | Parent Objet Type | +| `recording_url` | text | | Recording URL of the meeting. | +| `scheduled_date` | timestamp | ✔︎ | Time at which meeting was scheduled to start. | +| `state` | enum | ✔︎ | The state of meeting. | +| `tags` | reference (collection) -> [#record:tag] | | Tags associated with the meeting. | +| `title` | text | ✔︎ | Title of the meeting object. | + + +[ ▲ top](#summary) + +--- + +## microservice + +**Resource Type:** `part`, `microservice` + +**Capabilities:** Can Extract, Can Load, Can Subtype + +### Fixed Fields + +| Field | Value | +|-------|-------| +| `part_type` | "microservice" | + +### Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `created_by_id` | reference -> [#category:user] | | User ID of the user that created the object | +| `delivered_as` | enum (collection) | | Methods the product can be delivered as | +| `description` | rich_text | | Description of the part | +| `development_owner_id` | reference -> [#record:devu] | | User ID of the development owner of the part | +| `docs` | struct (collection) | | Docs associated with the part | +| `item_url_field` | text | | Link to the item in the origin system | +| `modified_by_id` | reference -> [#category:user] | | User ID of the user that last modified the object | +| `name` | text | ✔︎ | Name of the part | +| `owned_by_ids` | reference (collection) -> [#record:devu] | ✔︎ | User IDs of the users that own the part | +| `parent_part` | reference -> [#category:part] | | Parent of this part | +| `pm_owner_id` | reference -> [#record:devu] | | User ID of the PM owner of the part | +| `qa_owner_id` | reference -> [#record:devu] | | User ID of the QA owner of the part | +| `ref_url` | text | | URL to the part details (git url, website, etc.) | +| `tags` | reference (collection) -> [#record:tag] | | Tags associated with the object | + + +[ ▲ top](#summary) + +--- + +## object_member + +**Resource Type:** `object_member` + +**Capabilities:** Can Load + +### Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `conditional_role_id` | reference -> [#record:conditional_role] | | Conditional Role that entails the permissions of the member | +| `created_by_id` | reference -> [#category:user] | | User ID of the user that created the object | +| `member_id` | reference -> [#category:user] | ✔︎ | ID of the user or group | +| `object_id` | reference -> [#record:group] | ✔︎ | Globally unique DevRev Object Name (DON) for the object where the member is being added. | +| `role_id` | reference -> [#record:role] | | Role that entails the permissions of the member | +| `target_object_type` | enum | ✔︎ | Type of target object | +| `valid_from_date` | timestamp | ✔︎ | Timestamp when this membership is valid | +| `valid_to_date` | timestamp | | Timestamp when this membership expires | + + +[ ▲ top](#summary) + +--- + +## opportunity + +**Resource Type:** `work`, `opportunity` + +**Capabilities:** Can Extract, Can Load, Can Subtype + +### Fixed Fields + +| Field | Value | +|-------|-------| +| `work_type` | "opportunity" | + +### Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `account_id` | reference -> [#record:account] | | ID of the account that the opportunity is related to. | +| `actual_close_date` | timestamp | | Timestamp when the work was actually completed | +| `amount` | float | | Total opportunity amount. | +| `annual_contract_value` | struct | | Annual contract value of the opportunity. | +| `annual_recurring_revenue` | float | | Annual recurring revenue of the opportunity. | +| `body` | rich_text | | Body of the work object | +| `budget` | struct | | Budget of the opportunity. | +| `contact_ids` | reference (collection) -> [#record:revu] | | Contacts involved in the opportunity. | +| `created_by_id` | reference -> [#category:user] | | User ID of the user that created the object | +| `customer_budget` | float | | Budget of the customer. | +| `fallback_parts` | reference -> [#category:part] | | Other values that could serve as part if apples_to_part_id is found to be invalid. | +| `forecast_category` | enum | ✔︎ | Forecast category of the opportunity. | +| `item_url_field` | text | | Link to the item in the origin system | +| `modified_by_id` | reference -> [#category:user] | | User ID of the user that last modified the object | +| `owned_by_ids` | reference (collection) -> [#category:user] | ✔︎ | User IDs of the users that own the work | +| `priority` | enum | ✔︎ | Priority of the opportunity. | +| `probability` | float | | Probability of winning the deal, value lies between 0.0% and 100.0% | +| `reported_by_ids` | reference (collection) -> [#category:user] | | User IDs of the users that reported the work | +| `stage` | enum | ✔︎ | Stage of the object | +| `tags` | reference (collection) -> [#record:tag] | | Tags associated with the object | +| `target_close_date` | timestamp | | Timestamp when the work is expected to be complete | +| `title` | text | ✔︎ | Title of the work object | +| `value` | struct | | Value of the opportunity. | + +### Structs + +#### money + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `amount` | text | ✔︎ | Amount of the money. | +| `currency` | text | ✔︎ | Currency of the money. | + + +[ ▲ top](#summary) + +--- + +## product + +**Resource Type:** `part`, `product` + +**Capabilities:** Can Extract, Can Load, Can Subtype + +### Fixed Fields + +| Field | Value | +|-------|-------| +| `part_type` | "product" | + +### Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `created_by_id` | reference -> [#category:user] | | User ID of the user that created the object | +| `delivered_as` | enum (collection) | | Methods the product can be delivered as | +| `description` | rich_text | | Description of the part | +| `docs` | struct (collection) | | Docs associated with the part | +| `fulfilled_by` | reference (collection) -> [#category:part] | | IDs of the runnables that fulfill this product | +| `item_url_field` | text | | Link to the item in the origin system | +| `modified_by_id` | reference -> [#category:user] | | User ID of the user that last modified the object | +| `name` | text | ✔︎ | Name of the part | +| `owned_by_ids` | reference (collection) -> [#record:devu] | ✔︎ | User IDs of the users that own the part | +| `pm_owner_id` | reference -> [#record:devu] | | User ID of the PM owner of the part | +| `product_delivered_as` | enum | | Product Delivered as | +| `ref_url` | text | | URL to the part details (git url, website, etc.) | +| `tags` | reference (collection) -> [#record:tag] | | Tags associated with the object | + + +[ ▲ top](#summary) + +--- + +## revu + +**Resource Type:** `rev_user` + +**Capabilities:** Can Load + +### Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `created_by_id` | reference -> [#category:user] | | User ID of the user that created the object | +| `description` | rich_text | | Description of the Rev user. | +| `display_handle` | text | | The user's display handle. This field is deprecated now. Use display_name field instead. | +| `display_name` | text | ✔︎ | The user's display name. The name is non-unique and mutable. | +| `display_picture` | reference -> [#record:artifact] | | Artifact ID of the user's display picture | +| `email` | text | | Email address of the user. | +| `external_uid` | text | | External ref is a mutable unique identifier for a user within the Rev organization from your primary customer record. If none is available, a good alternative is the email address/phone number which could uniquely identify the user. If none is specified, a system-generated identifier will be assigned to the user. | +| `full_name` | text | | Full name of the user. | +| `phone_numbers` | text (collection) | | Phone numbers of the user. | +| `rev_org` | reference -> [#record:account #record:revo] | | The ID of the parent Rev Organization. | +| `state` | enum | | State of the user. | +| `tags` | reference (collection) -> [#record:tag] | | Tags associated with the object | +| `user_type` | enum | | Type of the user. | + + +[ ▲ top](#summary) + +--- + +## runnable + +**Resource Type:** `part`, `runnable` + +**Capabilities:** Can Extract, Can Load, Can Subtype + +### Fixed Fields + +| Field | Value | +|-------|-------| +| `part_type` | "runnable" | + +### Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `coderepo_paths` | text | | Paths in the repository for the code part | +| `coderepo_url` | text | | URL to the server & repo for the code part | +| `created_by_id` | reference -> [#category:user] | | User ID of the user that created the object | +| `delivered_as` | enum (collection) | | Methods the product can be delivered as | +| `description` | rich_text | | Description of the part | +| `discoveryevidences` | text (collection) | | A evidences that the inferer were able to find that justify the inference outcome | +| `docs` | struct (collection) | | Docs associated with the part | +| `item_url_field` | text | | Link to the item in the origin system | +| `kind` | enum | | The kind of runnable | +| `modified_by_id` | reference -> [#category:user] | | User ID of the user that last modified the object | +| `name` | text | ✔︎ | Name of the part | +| `owned_by_ids` | reference (collection) -> [#record:devu] | ✔︎ | User IDs of the users that own the part | +| `parent_part` | reference -> [#category:part] | | Parent of this part | +| `ref_url` | text | | URL to the part details (git url, website, etc.) | +| `tags` | reference (collection) -> [#record:tag] | | Tags associated with the object | + + +[ ▲ top](#summary) + +--- + +## sysu + +**Resource Type:** `sys_user` + +**Capabilities:** Can Load + +### Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `display_name` | text | ✔︎ | The user's display name. The name is non-unique and mutable. | +| `full_name` | text | | Full name of the user. | +| `state` | enum | | State of the user. | + + +[ ▲ top](#summary) + +--- + +## tag + +**Resource Type:** `tag` + +**Capabilities:** Can Load + +### Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `access_level` | enum | | Access level for the tag | +| `created_by_id` | reference -> [#category:user] | | User ID of the user that created the object | +| `description` | rich_text | | Description of the tag | +| `name` | text | ✔︎ | Name of the tag | +| `style` | text | | hex color of in the format #RRGGBB | + + +[ ▲ top](#summary) + +--- + +## task + +**Resource Type:** `work`, `task` + +**Capabilities:** Can Extract, Can Load, Can Subtype + +### Fixed Fields + +| Field | Value | +|-------|-------| +| `work_type` | "task" | + +### Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `actual_close_date` | timestamp | | Timestamp when the work was actually completed | +| `actual_effort` | float | | Actual effort to complete the task | +| `applies_to_part_id` | reference -> [#category:part] | | Details of the part relevant to the work | +| `applies_to_version_ids` | reference (collection) -> [#record:version] | | Part versions relevant to the work | +| `body` | rich_text | | Body of the work object | +| `created_by_id` | reference -> [#category:user] | | User ID of the user that created the object | +| `embedded` | bool | | Whether this task is an embedded task of another work or not | +| `estimated_effort` | float | | Estimated effort to complete the task | +| `fallback_parts` | reference -> [#category:part] | | Other values that could serve as part if apples_to_part_id is found to be invalid. | +| `item_url_field` | text | | Link to the item in the origin system | +| `modified_by_id` | reference -> [#category:user] | | User ID of the user that last modified the object | +| `owned_by_ids` | reference (collection) -> [#category:user] | ✔︎ | User IDs of the users that own the work | +| `priority` | enum | | Priority of the work based upon impact and criticality | +| `reported_by_ids` | reference (collection) -> [#category:user] | | User IDs of the users that reported the work | +| `stage` | enum | ✔︎ | Stage of the object | +| `start_date` | timestamp | | Timestamp when the task was started | +| `tags` | reference (collection) -> [#record:tag] | | Tags associated with the object | +| `target_close_date` | timestamp | | Timestamp when the work is expected to be complete | +| `title` | text | ✔︎ | Title of the work object | + + +[ ▲ top](#summary) + +--- + +## test_custom_type + +**Resource Type:** `custom_object` + +**Capabilities:** Can Load, Can Subtype + +### Fixed Fields + +| Field | Value | +|-------|-------| +| `leaf_type` | "test_custom_type" | + +### Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `created_by_id` | reference -> [#category:user] | | User ID of the user that created the object | +| `modified_by_id` | reference -> [#category:user] | | User ID of the user that last modified the object | +| `title` | text | | Title of the custom object | + + +[ ▲ top](#summary) + +--- + +## ticket + +**Resource Type:** `work`, `ticket` + +**Capabilities:** Can Extract, Can Load, Can Subtype + +### Fixed Fields + +| Field | Value | +|-------|-------| +| `work_type` | "ticket" | + +### Fields + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `account_id` | reference -> [#record:account] | | Globally unique ID for a DevRev account | +| `actual_close_date` | timestamp | | Timestamp when the work was actually completed | +| `applies_to_part_id` | reference -> [#category:part] | ✔︎ | Details of the part relevant to the work | +| `applies_to_version_ids` | reference (collection) -> [#record:version] | | Part versions relevant to the work | +| `body` | rich_text | | Body of the work object | +| `channels` | enum (collection) | | Channels of the ticket | +| `channels_v2` | reference (collection) -> [#record:external_communication_channel] | | Channel IDs of the ticket | +| `created_by_id` | reference -> [#category:user] | | User ID of the user that created the object | +| `fallback_parts` | reference -> [#category:part] | | Other values that could serve as part if apples_to_part_id is found to be invalid. | +| `group` | reference -> [#record:group] | | Group that owns the ticket | +| `is_spam` | bool | | Whether the ticket is spam. | +| `item_url_field` | text | | Link to the item in the origin system | +| `modified_by_id` | reference -> [#category:user] | | User ID of the user that last modified the object | +| `needs_response` | bool | | Whether the ticket needs a response. | +| `owned_by_ids` | reference (collection) -> [#category:user] | ✔︎ | User IDs of the users that own the work | +| `reported_by_ids` | reference (collection) -> [#category:user] | | User IDs of the users that reported the work | +| `rev_oid` | reference -> [#record:account #record:revo] | | Globally unique ID for a DevRev rev org | +| `sentiment` | enum | | Sentiment of the object | +| `sentiment_modified_date` | timestamp | | Timestamp when the sentiment was last modified | +| `sentiment_summary` | text | | Summary justifying the sentiment | +| `severity` | enum | ✔︎ | Severity of the ticket | +| `sla_id` | reference -> [#record:sla] | | The ID of the SLA applying to this object, controlling how the SLA tracking service treats it. | +| `sla_tracker_id` | reference -> [#record:sla_tracker] | | The ID of the SLA tracker applied to this object, which is used to track the SLA metrics. | +| `source_channel_v2` | reference -> [#record:external_communication_channel] | | Source channel ID of the ticket | +| `stage` | enum | ✔︎ | Stage of the object | +| `tags` | reference (collection) -> [#record:tag] | | Tags associated with the object | +| `target_close_date` | timestamp | | Timestamp when the work is expected to be complete | +| `title` | text | ✔︎ | Title of the work object | +| `visibility` | enum | | The visibility of the object. If 'internal', the object is visible within the Dev organization. If 'external', it is visible to both the Dev organization and Rev users. If not set, the default visibility is 'external'. | + + +[ ▲ top](#summary) + +--- \ No newline at end of file diff --git a/fern/versions/public.yml b/fern/versions/public.yml index b94354e7..e8a80105 100644 --- a/fern/versions/public.yml +++ b/fern/versions/public.yml @@ -195,6 +195,9 @@ navigation: - page: "Common issues" slug: faq path: ../docs/pages/airdrop/faq.mdx + - page: "Supported DevRev object types" + slug: supported-object-types + path: ../docs/pages/airdrop/supported-object-types.mdx - section: Data model slug: data-model contents: