Skip to content

Commit

Permalink
snowflake native apps: updating docs w/ scheduled syncs and some clea…
Browse files Browse the repository at this point in the history 
…nup ahead of public preview (#31853)
  • Loading branch information
@brianjlai
brianjlai committed Oct 27, 2023
1 parent b8ad0c6 commit 7e664bc
Show file tree
Hide file tree
Showing 4 changed files with 97 additions and 49 deletions.
Binary file added docs/snowflake-native-apps/facebook-marketing-shared-application.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
72 changes: 50 additions & 22 deletions docs/snowflake-native-apps/facebook-marketing.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,11 @@ Do not refresh the Apps page while the application is being installed. This may
2. On the left sidebar, click `Marketplace`.
3. Search for `Facebook Marketing Connector` by Airbyte or navigate to https://app.snowflake.com/marketplace/listing/GZTYZ9BCRT8/airbyte-facebook-marketing-connector-by-airbyte
4. Click `Request`. This will send a request that we will manually service as soon as we can.

![](./facebook-marketing-shared-application.png)

5. On the left sidebar, click `Apps`.
6. Under the `Recently Shared with You` section, you should see the `Facebook Marketing Connector by Airbyte`. Click `Get`.
6. Under the `Recently Shared with You` section, you should see the `Airbyte Facebook Marketing Connector` by Airbyte. Click `Get`.
7. Expand `Options`.
1. You can rename the application or leave the default. This is how you will reference the application from a worksheet.
2. Specify the warehouse that the application will be installed to.
Expand All @@ -37,23 +40,23 @@ In order for the Facebook Marketing Connector by Airbyte to query Facebook's API
## Snowflake Native App Authorizations

:::note
By default the app will be installed using the name `AIRBYTE_FACEBOOK_MARKETING`, but if you renamed the app during installation, you will have to use that name as a reference.
By default the app will be installed using the name `AIRBYTE_FACEBOOK_MARKETING_CONNECTOR`, but if you renamed the app during installation, you will have to use that name as a reference.
:::

1. Create the database where the app will access the authorization.
```
CREATE DATABASE airbyte_facebook_marketing_db;
USE airbyte_facebook_marketing_db;
CREATE DATABASE AIRBYTE_FACEBOOK_MARKETING_DB ;
USE AIRBYTE_FACEBOOK_MARKETING_DB;
```

2. The native app will validate the output database and create it if it does not exist. In order to do that, the app needs access to the database:
```
GRANT CREATE DATABASE ON ACCOUNT TO APPLICATION <app_name>;
GRANT CREATE DATABASE ON ACCOUNT TO APPLICATION AIRBYTE_FACEBOOK_MARKETING_CONNECTOR;
```

3. You will need to allow outgoing network traffic based on the domain of the source. In the case of Facebook Marketing, simply run:
```
CREATE OR REPLACE NETWORK RULE facebook_marketing_apis_network_rule
CREATE OR REPLACE NETWORK RULE FACEBOOK_MARKETING_APIS_NETWORK_RULE
MODE = EGRESS
TYPE = HOST_PORT
VALUE_LIST = ('graph.facebook.com');
Expand All @@ -65,7 +68,7 @@ As of 2023-09-13, the [Snowflake documentation](https://docs.snowflake.com/en/sq

4. Once you have external access configured, you need define your authorization/authentication. Provide the credentials to the app as such:
```
CREATE OR REPLACE SECRET airbyte_app_secret
CREATE OR REPLACE SECRET AIRBYTE_APP_SECRET
TYPE = GENERIC_STRING
SECRET_STRING = '{
"access_token": "<access_token>"
Expand All @@ -75,26 +78,32 @@ CREATE OR REPLACE SECRET airbyte_app_secret

5. Once the network rule and the secret are defined in Snowflake, you need to make them available to the app by using an external access integration.
```
CREATE OR REPLACE EXTERNAL ACCESS INTEGRATION airbyte_app_integration
CREATE OR REPLACE EXTERNAL ACCESS INTEGRATION AIRBYTE_APP_INTEGRATION
ALLOWED_NETWORK_RULES = (facebook_marketing_apis_network_rule)
ALLOWED_AUTHENTICATION_SECRETS = (airbyte_app_secret)
ALLOWED_AUTHENTICATION_SECRETS = (AIRBYTE_APP_SECRET)
ENABLED = true;
```

6. Grant permission for the app to access the integration.
```
GRANT USAGE ON INTEGRATION airbyte_app_integration TO APPLICATION AIRBYTE_FACEBOOK_MARKETING;
GRANT USAGE ON INTEGRATION AIRBYTE_APP_INTEGRATION TO APPLICATION AIRBYTE_FACEBOOK_MARKETING_CONNECTOR;
```

7. Grant permissions for the app to access the database that houses the secret and read the secret.
```
GRANT USAGE ON DATABASE airbyte_facebook_marketing_db TO APPLICATION AIRBYTE_FACEBOOK_MARKETING;
GRANT USAGE ON SCHEMA public TO APPLICATION AIRBYTE_FACEBOOK_MARKETING;
GRANT READ ON SECRET airbyte_app_secret TO APPLICATION AIRBYTE_FACEBOOK_MARKETING;
GRANT USAGE ON DATABASE AIRBYTE_FACEBOOK_MARKETING_DB TO APPLICATION AIRBYTE_FACEBOOK_MARKETING_CONNECTOR;
GRANT USAGE ON SCHEMA PUBLIC TO APPLICATION AIRBYTE_FACEBOOK_MARKETING_CONNECTOR;
GRANT READ ON SECRET AIRBYTE_APP_SECRET TO APPLICATION AIRBYTE_FACEBOOK_MARKETING_CONNECTOR;
```

8. Grant permissions for the app to create a warehouse on which to execute sync tasks, and to execute tasks.
```
GRANT CREATE WAREHOUSE ON ACCOUNT TO APPLICATION AIRBYTE_FACEBOOK_MARKETING_CONNECTOR;
GRANT EXECUTE TASK ON ACCOUNT TO APPLICATION AIRBYTE_FACEBOOK_MARKETING_CONNECTOR;
```


## Configure a connection
## Configure a Connection
Once this is all set up, you can now configure a connection. To do so, use the Streamlit app by going in the `Apps` section and selecting `AIRBYTE_FACEBOOK_MARKETING`. You will have to accept the Anaconda terms in order to use Streamlit.

Once you have access to the app, select `New Connection` and fill the following fields:
Expand All @@ -109,13 +118,13 @@ The Facebook Ad account ID to use when pulling data from the Facebook Marketing

`start_date`

UTC date in the format 2021-09-29T12:13:14Z. Any data before this date will not be replicated.
UTC date in the format YYYY-MM-DDTHH:mm:ssZ (e.g. 2021-09-29T12:13:14Z). Any data before this date will not be replicated.

---

`end_date`

UTC date in the format 2021-09-29T12:13:14Z. Any data after this date will not be replicated.
UTC date in the format YYYY-MM-DDTHH:mm:ssZ (e.g. 2021-09-29T12:13:14Z). Any data after this date will not be replicated.

---

Expand Down Expand Up @@ -161,25 +170,44 @@ The table where the schema will be saved. Snowflake's table [naming convention](

---

`Connection name`
`Connection Name`

How the connection will be referred in the Streamlit app.

---

`Replication Frequency`

How often records are fetched.
The sync schedule that determines how often your data will be synced to the target database.

---

## Enabling Logging and Event Sharing for an Application
Sharing the logging and telemetry data of your installed application helps us improve the application and can allow us to better triage problems that your run into. To configure your application for logging and telemetry data please refer to the documentation for [Enabling Logging and Event Sharing](event-sharing.md).

## Run a sync
Once a connection is configured, go in `Connections List` and click on `Sync Now` for the connection you want to sync. Once the sync is complete, you should be able to validate that the records have been stored in `<your_database>.<your_schema>`

### Supported Streams
## Syncing Your Facebook Marketing Data
Once a connection is configured, go in `Connections List` to view all of your connections. From here for each connection you can
view the configuration settings, start a sync, and view the prior sync history.

### Scheduled Syncs
While creating a connection, you can specify a "Replication Frequency" which will dictate how often your data will be extracted from
Facebook Marketing and loaded into your Snowflake database. This process is started automatically according to your schedule and does not
require that you manually trigger syncs. For example, if you create a connection at 10:15 AM and set your replication frequency to
hourly, then a sync will be started immediately. The next sync will start at 11:15 AM onwards. Only one sync is active at the same
time. In the event that your sync runs longer than one hour, a new sync will start at the next available time.

### Manual Syncs
In addition to scheduled syncs, you can also configure a connection to only sync data on-demand by setting "Replication Frequency" to
`MANUAL`. After creating a connection, from the `Connections List` page, you can use the "Sync Now" button to trigger a sync of
your API data to your Snowflake database. You can also use this button to manually trigger connections that sync according to a
schedule. If there is already a sync in progress, this button will be disabled.

### Sync History
From the `Connections List` page, you can view information about past syncs for each connection to determine when your
data is done syncing and whether the operation was successful. Once the sync is completed successfully, you should be
able to validate that the records have been stored in `<your_database>.<your_schema>`.

## Supported Streams
As of now, all supported streams perform a full refresh. Incremental syncs are not yet supported. Here are the list of supported streams:
* Activities
* Ad Account
Expand Down
Binary file added docs/snowflake-native-apps/linkedin-ads-shared-application.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
74 changes: 47 additions & 27 deletions docs/snowflake-native-apps/linkedin-ads.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,11 @@ Do not refresh the Apps page while the application is being installed. This may
2. On the left sidebar, click `Marketplace`.
3. Search for `Linkedin Ads Connector` by Airbyte or navigate to https://app.snowflake.com/marketplace/listing/GZTYZ9BCRSJ/airbyte-linkedin-ads-connector-by-airbyte
4. Click `Request`. This will send a request that we will manually service as soon as we can.

![](./linkedin-ads-shared-application.png)

5. On the left sidebar, click `Apps`.
6. Under the `Recently Shared with You` section, you should see the `Linkedin Ads Connector by Airbyte`. Click `Get`.
6. Under the `Recently Shared with You` section, you should see the `Airbyte LinkedIn Ads Connector` by Airbyte. Click `Get`.
7. Expand `Options`.
1. You can rename the application or leave the default. This is how you will reference the application from a worksheet.
2. Specify the warehouse that the application will be installed to.
Expand All @@ -37,23 +40,23 @@ In order for the LinkedIn Ads Connector by Airbyte to query LinkedIn, you will n
## Snowflake Native App Authorizations

:::note
By default the app will be installed using the name `AIRBYTE_LINKEDIN_ADS`, but if you renamed the app during installation, you will have to use that name as a reference.
By default the app will be installed using the name `AIRBYTE_LINKEDIN_ADS_CONNECTOR`, but if you renamed the app during installation, you will have to use that name as a reference.
:::

1. Create the database where the app will access the authorization.
```
CREATE DATABASE airbyte_linkedin_ads_db;
USE airbyte_linkedin_ads_db;
CREATE DATABASE AIRBYTE_LINKEDIN_ADS_DB;
USE AIRBYTE_LINKEDIN_ADS_DB;
```

2. The native app will validate the output database and create it if it does not exist. In order to do that, the app needs access to the database:
```
GRANT CREATE DATABASE ON ACCOUNT TO APPLICATION <app_name>;
GRANT CREATE DATABASE ON ACCOUNT TO APPLICATION AIRBYTE_LINKEDIN_ADS_CONNECTOR;
```

3. You will need to allow outgoing network traffic based on the domain of the source. In the case of LinkedIn Ads, simply run:
```
CREATE OR REPLACE NETWORK RULE linkedin_apis_network_rule
CREATE OR REPLACE NETWORK RULE LINKEDIN_APIS_NETWORK_RULE
MODE = EGRESS
TYPE = HOST_PORT
VALUE_LIST = ('api.linkedin.com', 'www.linkedin.com', 'linkedin.com');
Expand All @@ -65,7 +68,7 @@ As of 2023-09-13, the [Snowflake documentation](https://docs.snowflake.com/en/sq

4. Once you have external access configured, you need define your authorization/authentication. Provide the credentials to the app as such:
```
CREATE OR REPLACE SECRET airbyte_app_secret
CREATE OR REPLACE SECRET AIRBYTE_APP_SECRET
TYPE = GENERIC_STRING
SECRET_STRING = '{
"auth_method": "oAuth2.0",
Expand All @@ -78,43 +81,41 @@ CREATE OR REPLACE SECRET airbyte_app_secret

5. Once the network rule and the secret are defined in Snowflake, you need to make them available to the app by using an external access integration.
```
CREATE OR REPLACE EXTERNAL ACCESS INTEGRATION airbyte_app_integration
ALLOWED_NETWORK_RULES = (linkedin_apis_network_rule)
ALLOWED_AUTHENTICATION_SECRETS = (airbyte_app_secret)
CREATE OR REPLACE EXTERNAL ACCESS INTEGRATION AIRBYTE_APP_INTEGRATION
ALLOWED_NETWORK_RULES = (LINKEDIN_APIS_NETWORK_RULE)
ALLOWED_AUTHENTICATION_SECRETS = (AIRBYTE_APP_SECRET)
ENABLED = true;
```

6. Grant permission for the app to access the integration.
```
GRANT USAGE ON INTEGRATION airbyte_app_integration TO APPLICATION AIRBYTE_LINKEDIN_ADS;
GRANT USAGE ON INTEGRATION AIRBYTE_APP_INTEGRATION TO APPLICATION AIRBYTE_LINKEDIN_ADS_CONNECTOR;
```

7. Grant permissions for the app to access the database that houses the secret and read the secret.
```
GRANT USAGE ON DATABASE airbyte_linkedin_ads_db TO APPLICATION AIRBYTE_LINKEDIN_ADS;
GRANT USAGE ON SCHEMA public TO APPLICATION AIRBYTE_LINKEDIN_ADS;
GRANT READ ON SECRET airbyte_app_secret TO APPLICATION AIRBYTE_LINKEDIN_ADS;
GRANT USAGE ON DATABASE AIRBYTE_LINKEDIN_ADS_DB TO APPLICATION AIRBYTE_LINKEDIN_ADS_CONNECTOR;
GRANT USAGE ON SCHEMA PUBLIC TO APPLICATION AIRBYTE_LINKEDIN_ADS_CONNECTOR;
GRANT READ ON SECRET AIRBYTE_APP_SECRET TO APPLICATION AIRBYTE_LINKEDIN_ADS_CONNECTOR;
```

8. Grant permissions for the app to create a warehouse on which to execute sync tasks, and to execute tasks.
```
GRANT CREATE WAREHOUSE ON ACCOUNT TO APPLICATION AIRBYTE_LINKEDIN_ADS;
GRANT EXECUTE TASK ON ACCOUNT TO APPLICATION AIRBYTE_LINKEDIN_ADS;
GRANT CREATE WAREHOUSE ON ACCOUNT TO APPLICATION AIRBYTE_LINKEDIN_ADS_CONNECTOR;
GRANT EXECUTE TASK ON ACCOUNT TO APPLICATION AIRBYTE_LINKEDIN_ADS_CONNECTOR;
```


## Configure a connection
Once this is all set up, you can now configure a connection. To do so, use the Streamlit app by going in the `Apps` section and selecting `AIRBYTE_LINKEDIN_ADS`. You will have to accept the Anaconda terms in order to use Streamlit.

![](./linkedin-ads-ui.gif)
## Configuring a Connection
Once this is all set up, you can now configure a connection. To do so, use the Streamlit app by going in the `Apps` section and selecting `AIRBYTE_LINKEDIN_ADS`. You will have to accept the Anaconda terms in order to use Streamlit.

Once you have access to the app, select `New Connection` and fill the following fields:

---

`start_date`

UTC date in the format 2020-09-17. Any data before this date will not be replicated.
UTC date in the format YYYY-MM-DD (e.g. 2020-09-17). Any data before this date will not be replicated.

---

Expand All @@ -136,25 +137,44 @@ The table where the schema will be saved. Snowflake's table [naming convention](

---

`Connection name`
`Connection Name`

How the connection will be referred in the Streamlit app.

---

`Replication Frequency`

How often records are fetched.
The sync schedule that determines how often your data will be synced to the target database.

---

## Enabling Logging and Event Sharing for an Application
Sharing the logging and telemetry data of your installed application helps us improve the application and can allow us to better triage problems that your run into. To configure your application for logging and telemetry data please refer to the documentation for [Enabling Logging and Event Sharing](event-sharing.md).

## Run a sync
Once a connection is configured, go in `Connections List` and click on `Sync Now` for the connection you want to sync. Once the sync is complete, you should be able to validate that the records have been stored in `<your_database>.<your_schema>`

### Supported Streams
## Syncing Your LinkedIn Ads Data
Once a connection is configured, go in `Connections List` to view all of your connections. From here for each connection you can
view the configuration settings, start a sync, and view the prior sync history.

### Scheduled Syncs
While creating a connection, you can specify a "Replication Frequency" which will dictate how often your data will be extracted from
LinkedIn Ads and loaded into your Snowflake database. This process is started automatically according to your schedule and does not
require that you manually trigger syncs. For example, if you create a connection at 10:15 AM and set your replication frequency to
hourly, then a sync will be started immediately. The next sync will start at 11:15 AM onwards. Only one sync is active at the same
time. In the event that your sync runs longer than one hour, a new sync will start at the next available time.

### Manual Syncs
In addition to scheduled syncs, you can also configure a connection to only sync data on-demand by setting "Replication Frequency" to
`MANUAL`. After creating a connection, from the `Connections List` page, you can use the "Sync Now" button to trigger a sync of
your API data to your Snowflake database. You can also use this button to manually trigger connections that sync according to a
schedule. If there is already a sync in progress, this button will be disabled.

### Sync History
From the `Connections List` page, you can view information about past syncs for each connection to determine when your
data is done syncing and whether the operation was successful. Once the sync is completed successfully, you should be
able to validate that the records have been stored in `<your_database>.<your_schema>`.

## Supported Streams
As of now, all supported streams perform a full refresh. Incremental syncs are not yet supported. Here are the list of supported streams:
* Accounts
* Account Users
Expand Down

0 comments on commit 7e664bc

Please sign in to comment.