Skip to content

Commit

Permalink
Restore cloud markdown files for version 0.17
Browse files Browse the repository at this point in the history
  • Loading branch information
JessSaavedra committed Feb 21, 2024
1 parent 057cefc commit 8768f3a
Show file tree
Hide file tree
Showing 10 changed files with 770 additions and 0 deletions.
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
---
sidebar_label: 'Manage Checkpoints'
title: 'Manage Checkpoints'
description: Create and manage Checkpoints in GX Cloud.
---

A Checkpoint validates Expectation Suite data. After you create a Checkpoint to validate data, you can save and reuse the Checkpoint.

To learn more about Checkpoints, see [Checkpoint](../../terms/checkpoint.md).

## Prerequisites

- You have [set up your environment](../set_up_gx_cloud.md) and the GX Agent is running.

- You have a [Data Asset](/docs/cloud/data_assets/manage_data_assets#create-a-data-asset).

- You have created an [Expectation](/docs/cloud/expectations/manage_expectations#create-an-expectation).

## Add a Checkpoint

1. In Jupyter Notebook, run the following code to import the `great_expectations` module and the existing Data Context:

```python title="Jupyter Notebook"
import great_expectations as gx
context = gx.get_context()
```
2. Run the following code to retrieve the Expectation Suite:

```python title="Jupyter Notebook"
expectation_suite = context.get_expectation_suite(expectation_suite_name=<expectation_name>)
```

3. Run the following code to assign a name to the Checkpoint:

```python title="Jupyter Notebook"
checkpoint_name = <checkpoint_name>
```

4. Run the following code to define the Checkpoint configuration including the Data Source and Data Asset names:

```python title="Jupyter Notebook"
checkpoint_config = {
"name": checkpoint_name,
"validations": [{
"expectation_suite_name": expectation_suite.expectation_suite_name,
"expectation_suite_ge_cloud_id": expectation_suite.ge_cloud_id,
"batch_request": {
"datasource_name": "<data_source_name>",
"data_asset_name": "<data_asset_name>",
},
}],
}
```
5. Run the following code to add the Checkpoint:

```python title="Jupyter Notebook"
checkpoint = context.add_or_update_checkpoint(**checkpoint_config)
```

6. Optional. Run the following code to confirm the Checkpoint name:

```python title="Jupyter Notebook"
print(checkpoint)
```

7. Optional. Run the following code to run the Checkpoint:

```python title="Jupyter Notebook"
result = checkpoint.run()
```

## Run a Checkpoint

1. In GX Cloud, click **Checkpoints**.

2. Optional. To run a Checkpoint on a failing Checkpoint, click **Failures Only**.

3. Optional. To run a specific Checkpoint, select it in the **Checkpoints** pane.

4. Click **Run Checkpoint** for the Checkpoint you want to run.


## Edit a Checkpoint configuration

1. In Jupyter Notebook, run the following code to import the `great_expectations` module and the existing Data Context:

```python title="Jupyter Notebook"
import great_expectations as gx
context = gx.get_context()
```
2. Run the following Python code to retrieve the Checkpoint:

```python title="Jupyter Notebook"
retrieved_checkpoint = context.get_checkpoint(name="version-0.17.23 <checkpoint_name>")
```
3. Edit the Checkpoint configuration.

4. Run the following code to update the Checkpoint configuration:

```python title="Jupyter Notebook"
checkpoint = context.add_or_update_checkpoint(**checkpoint_config)
```

## Delete a Checkpoint

1. In GX Cloud, click **Checkpoints**.

2. Click **Delete Checkpoint** for the Checkpoint you want to delete.

3. Click **Delete**.
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
---
sidebar_label: 'Manage Data Assets'
title: 'Manage Data Assets'
description: Create and manage Data Assets in GX Cloud.
toc_min_heading_level: 2
toc_max_heading_level: 2
---

A Data Asset is a collection of records that you create when you connect to your Data Source. When you connect to your Data Source, you define a minimum of one Data Asset. You use these Data Assets to create the Batch Requests that select the data that is provided to your Expectations.

To learn more about Data Assets, see [Data Asset](../../terms/data_asset.md).

## Prerequisites

- You have a [GX Cloud Beta account](https://greatexpectations.io/cloud).

- You have [set up your environment](../set_up_gx_cloud.md), you have set the `GX_CLOUD_SNOWFLAKE_PASSWORD` environment variable, and the GX Agent is running.

- You have a [Snowflake account](https://docs.snowflake.com/en/user-guide-admin) with USAGE privileges on the table, database, and schema you are validating, and you know your password.


## Create a Data Asset

Create a Data Asset to define the data you want GX Cloud to access. Currently, the GX Cloud user interface is configured for Snowflake. To connect to Data Assets on another Data Source, see [Connect to source data](https://deploy-preview-8760.docs.greatexpectations.io/docs/guides/connecting_to_your_data/connect_to_data_lp) in the GX OSS documentation.

1. In GX Cloud, click **Data Assets** > **New Asset**.

2. Complete the following mandatory fields:

- **Datasource name**: Enter a meaningful name for the Data Asset.

- **Username**: Enter your Snowflake username.

- **Password variable**: Enter `GX_CLOUD_SNOWFLAKE_PASSWORD`.

- **Account or locator**: Enter your Snowflake account or locator information. The locator value must include the geographical region. For example, `us-east-1`. To locate these values see [Account Identifiers](https://docs.snowflake.com/en/user-guide/admin-account-identifier).

3. Optional. Complete the following fields:

- **Database**: Enter the name of the Snowflake database where the data you want to validate is stored.

- **Schema**: Enter the name of the schema for the Snowflake database where the data you want to validate is stored.

- **Warehouse**: Enter the name of the Snowflake database warehouse.

- **Role**: Enter your Snowflake role.

- **Authenticator**: Enter the Snowflake database authenticator that you want to use to verify your Snowflake connection.

4. Optional. Clear **Create temp table** if you don't want to create a temporary database table. Temporary database tables store data temporarily and can improve performance by making queries run faster.

5. Optional. Clear **Test connection** if you don't want to test the Data Asset connection. Testing the connection to the Data Asset is a preventative measure that makes sure the connection configuration is correct. This verification can help you avoid errors and can reduce troubleshooting downtime.

6. Select **Table Asset** or **Query Asset** and complete the following fields:

- **Asset name**: Enter a name for the Data Asset. Data Asset names must be unique. If you use the same name for multiple Data Assets, each Data Asset must be associated with a unique Data Source.

- **Table name**: When **Table Asset** is selected, enter a name for the table you're creating in the Data Asset.

- **Query**: When **Query Asset** is selected, enter the query that you want to run on the table.

7. Optional. Select **Add table/query** to add additional tables or queries and repeat step 6.

8. Click **Finish**.

9. Create an Expectation. See [Create an Expectation](/docs/cloud/expectations/manage_expectations#create-an-expectation).

## View Data Asset metrics

Data Asset metrics provide you with insight into the data you can use for your data validations.

1. In GX Cloud, click **Data Assets** and then select a Data Asset in the **Data Assets** list.

2. Click the **Overview** tab.

3. Select one of the following options:

- If you have not previously generated Data Asset metrics, click **Fetch Metrics**.

- If you previously generated Data Asset metrics, click **Refresh** to refresh the metrics.

### Available Data Asset metrics

The following table lists the available Data Asset metrics.

| Column | Description |
|------------------------------------------|-----------------------------------------------------------|
| **Row Count** | The number of rows within a Data Asset. |
| **Column** | A column within your Data Asset. |
| **Type** | The data storage type in the Data Asset column. |
| **Min** | For numeric columns the lowest value in the column. |
| **Max** | For numeric columns, the highest value in the column. |
| **Mean** | For numeric columns, the average value with the column.<br/> This is determined by dividing the sum of all values in the Data Asset by the number of values. |
| **Median** | For numeric columns, the value in the middle of a data set.<br/> 50% of the data within the Data Asset has a value smaller or equal to the median, and 50% of the data within the Data Asset has a value that is higher or equal to the median. |
| **Null %** | The percentage of missing values in a column. |


## Edit a Data Asset

1. In Jupyter Notebook, run the following code to import the `great_expectations` module and the existing Data Context:

```python title="Jupyter Notebook"
import great_expectations as gx
context = gx.get_context()
```

2. Run the following code to retrieve the Data Source:

```python title="Jupyter Notebook"
datasource = context.get_datasource("<data_source_name>")
```

3. Edit the Data Asset settings. For example, run the following code to change the name of the Data Source:

```python title="Jupyter Notebook"
datasource.name = "version-0.17.23 <new_data_source_name>"
```

To review the Data Asset parameters that you can add or edit, see the [GX API documentation](https://deploy-preview-8760.docs.greatexpectations.io/docs/reference/api_reference).

4. Run the following code to save your changes:

```python title="Jupyter Notebook"
context.sources.update_snowflake(datasource)
```

## Delete a Data Asset

1. In GX Cloud, click **Settings** > **Datasources**.

2. Click **Delete** for the Data Source and the associated Data Assets you want to delete.

3. Click **Delete**.

Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
---
sidebar_label: 'Manage Expectation Suites'
title: 'Manage Expectation Suites'
description: Create and manage Expectation Suites in GX Cloud.
---

Expectation Suites contain multiple Expectations for a single Data Asset. Like Expectations, they can help you better understand your data and help you improve data quality. A default Expectation Suite is created when you create a Data Asset.

To learn more about Expectation Suites, see [Expectation Suites](../../terms/expectation_suite.md).

## Prerequisites

- You have [set up your environment](../set_up_gx_cloud.md) and the GX Agent is running.

- You have a [Data Asset](/docs/cloud/data_assets/manage_data_assets#create-a-data-asset).

## Automatically create an Expectation Suite that tests for missing data

Automatically create an Expectation Suite that you can use to determine if your Data Asset contains missing data (null values). Creating Expectation Suites automatically saves you writing and then running the same code for each column in your Data Asset.

1. In GX Cloud, click **Data Assets** and select a Data Asset in the **Data Assets** list.

2. Click the **Expectations** tab.

3. Click **Create New Suite** in the **Expectation Suites** pane.

4. Click **Automatic (Experimental)**.

5. Click **Missingness** and then enter a name for the Expectation Suite in the **Suite name** field.

6. Click **Generate Expectations**.

It might take several minutes to create the Expectation Suite. When the process is complete, a new Expectation Suite appears in the **Expectation Suites** pane.

7. Optional. Run a Validation on the Expectation Suite. See [Run a Validation](/docs/cloud/validations/manage_validations#run-a-validation).

## Create an empty Expectation Suite

If you have specific business requirements, or you want to examine specific data, you can create an empty Expectation Suite and then add Expectations individually.

1. In GX Cloud, click **Data Assets** and select a Data Asset in the **Data Assets** list.

2. Click the **Expectations** tab.

3. Click **Create New Suite** in the **Expectation Suites** pane.

4. Click **Manual**.

5. Enter a name for the Expectation Suite in the **Suite name** field.

6. Click **Generate Expectations**.

7. Add Expectations to the Expectation Suite. See [Create an Expectation](/docs/cloud/expectations/manage_expectations#create-an-expectation).

8. Optional. Run a Validation on the Expectation Suite. See [Run a Validation](/docs/cloud/validations/manage_validations#run-a-validation).

## Delete an Expectation Suite

1. In GX Cloud, delete all Checkpoints associated with the Expectation Suite. See [Delete a Checkpoint](/docs/cloud/checkpoints/manage_checkpoints#delete-a-checkpoint).

2. Click **Expectation Suites**.

3. Click **Delete** for the Expectation Suite you want to delete.

4. Click **Delete**.

## Related documentation

- [Manage Expectations](../expectations/manage_expectations.md)

- [Manage Validations](../validations/manage_validations.md)
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
---
sidebar_label: 'Manage Expectations'
title: 'Manage Expectations'
description: Create and manage Expectations in GX Cloud.
---

An Expectation is a verifiable assertion about your data. They make implicit assumptions about your data explicit, and they provide a flexible, declarative language for describing expected behavior. They can help you better understand your data and help you improve data quality.

To learn more about Expectations, see [Expectation](../../terms/expectation.md).

## Prerequisites

- You have [set up your environment](../set_up_gx_cloud.md) and the GX Agent is running.

- You have a [Data Asset](/docs/cloud/data_assets/manage_data_assets#create-a-data-asset).

## Create an Expectation

1. In GX Cloud, click **Data Assets**.

2. In the **Data Assets** list, click the Data Asset name.

3. Click the **Expectations** tab.

4. Click **New Expectation**.

5. Select an Expectation type, enter the column name, and then complete the optional fields.

If you prefer to work in a code editor, or you want to configure an Expectation from the [Expectations Gallery](https://greatexpectations.io/expectations/), click the **JSON Editor** tab and define your Expectation parameters in the code pane.

6. Click **Save**.

7. Optional. Repeat steps 1 to 4 to add additional Expectations.

8. Optional. Run a Validation. See [Run a Validation](/docs/cloud/validations/manage_validations#run-a-validation).

## Edit an Expectation

1. In GX Cloud, click **Data Assets**.

2. In the **Data Assets** list, click the Data Asset name.

3. Click the **Expectations** tab.

4. Click **Edit Expectations** for the Expectation that you want to edit.

5. Edit the Expectation configuration.

If you prefer to work in a code editor, or you configured an Expectation from the [Expectations Gallery](https://greatexpectations.io/expectations/), click the **JSON Editor** tab and edit the Expectation parameters in the code pane.

6. Click **Save**.

## Delete an Expectation

1. In GX Cloud, click **Data Assets**.

2. In the **Data Assets** list, click the Data Asset name.

3. Click the **Expectations** tab.

4. Click **Delete Expectation** for the Expectation you want to delete.

5. Click **Yes, delete Expectation**.

## Related documentation

- [Manage Expectation Suites](../expectation_suites/manage_expectation_suites.md)

0 comments on commit 8768f3a

Please sign in to comment.