Skip to content

Commit

Permalink
Merge branch 'develop' into d/doc-655/checkpoint_guidance_for_1-0
Browse files Browse the repository at this point in the history
  • Loading branch information
@Rachel-Reverie
Rachel-Reverie committed Mar 29, 2024
2 parents 597aec6 + ab02092 commit 17d28e6
Show file tree
Hide file tree
Showing 71 changed files with 875 additions and 3,941 deletions.
9 changes: 9 additions & 0 deletions .git-blame-ignore-revs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -38,3 +38,12 @@ c3df7cde688332cb101432909145b37ff4108803
# black -> ruff format
# https://github.com/great-expectations/great_expectations/pull/9536
697d5910c8331721e7dc299b1c0a294de84275b5
# Ban unittest.mock usage
# https://github.com/great-expectations/great_expectations/pull/9586
fc5023b4e1f1dad2e9c6faa8897fa92991b821ab
# Reduce max cyclomatic complexity from 10 -> 8
# https://github.com/great-expectations/great_expectations/pull/9622
edf5dbdc77b418d659dc9d87462cd6df9a85436c
# Change line-length from 88 -> 100
# https://github.com/great-expectations/great_expectations/pull/9584
d170c89167a96b702edc02b16dbf5984619d0e8f
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -325,7 +325,7 @@ jobs:
- name: Run the tests
run:
invoke ci-tests 'cloud' --up-services --verbose --reports
invoke ci-tests 'cloud' --up-services --verbose --reports -W default::great_expectations.datasource.fluent.GxInvalidDatasourceWarning

# upload coverage report to codecov
- name: Upload coverage reports to Codecov
Expand Down
2 changes: 1 addition & 1 deletion assets/scripts/build_gallery.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
from typing import Dict, Final, List, Optional, Tuple

import click
import pkg_resources
import pkg_resources # noqa: TID251 # TODO: switch to importlib.metadata or importlib.resources

from great_expectations.compatibility import pydantic
from great_expectations.core.expectation_diagnostics.expectation_doctor import (
Expand Down
2 changes: 1 addition & 1 deletion docs/docusaurus/docs/cloud/connect/connect_airflow.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ Apache Airflow is an orchestration tool that allows you to schedule and monitor
def run_gx_airflow():

context = gx.get_context()
checkpoint = context.checkpoints.get(name = CHECKPOINT_NAME)
checkpoint = context.get_legacy_checkpoint(name=CHECKPOINT_NAME)
checkpoint.run()

default_args = {
Expand Down
145 changes: 2 additions & 143 deletions docs/docusaurus/docs/cloud/connect/connect_postgresql.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ New to GX Cloud and not sure that it's the right solution for your organization?

- You have a [GX Cloud account](https://greatexpectations.io/cloud) with [Admin or Editor permissions](../about_gx.md#roles-and-responsibilities).

- You have deployed the GX Agent. See [Deploy the GX Agent](../deploy_gx_agent.md).

- You have a PostgreSQL database, schema, and table.

- To improve data security, GX recommends creating a separate PostgreSQL user for your GX Cloud connection.
Expand Down Expand Up @@ -43,151 +45,8 @@ New to GX Cloud and not sure that it's the right solution for your organization?

4. Click **Execute/Refresh**.

## Get your GX Cloud access token and organization ID

You'll need your access token and organization ID to set your access credentials. Don't commit your access credentials to your version control software.

1. In GX Cloud, click **Settings** > **Tokens**.

2. In the **User access tokens** pane, click **Create user access token**.

3. In the **Token name** field, enter a name for the token that will help you quickly identify it.

4. Click **Create**.

5. Copy and then paste the user access token into a temporary file. The token can't be retrieved after you close the dialog.

6. Click **Close**.

7. Copy the value in the **Organization ID** field into the temporary file with your access token and then save the file.

GX recommends deleting the temporary file after you set the environment variables.

## Deploy the GX Agent

You deploy and run the GX Agent within your deployment environment. You can deploy the GX Agent container in any environment where you can run Docker container images.

To learn how to deploy a Docker container image in a specific environment, see the following documentation:

- [Quickstart: Deploy a container instance in Azure using the Azure CLI](https://learn.microsoft.com/en-us/azure/container-instances/container-instances-quickstart)

- [Build and push a Docker image with Google Cloud Build](https://cloud.google.com/build/docs/build-push-docker-image)

- [Deploy Docker Containers on Amazon ECS](https://aws.amazon.com/getting-started/hands-on/deploy-docker-containers/)

You can deploy the GX Agent in any environment in which you create Kubernetes clusters. For example:

- [Amazon Elastic Kubernetes Service (EKS)](https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html)

- [Microsoft Azure Kubernetes Service (AKS)](https://learn.microsoft.com/en-us/azure/architecture/reference-architectures/containers/aks-start-here)

- [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine/docs)

- Any Kubernetes cluster version 1.21 or greater which uses standard Kubernetes

<Tabs
groupId="connect-PostgreSQL"
defaultValue='docker'
values={[
{label: 'Docker', value:'docker'},
{label: 'Kubernetes', value:'kubernetes'},
]}>
<TabItem value="docker">

1. Download the GX Agent Docker container image from [Docker Hub](https://hub.docker.com/r/greatexpectations/agent).

2. After configuring your cloud service to run Docker containers, run the following Docker command to start the GX Agent:

```bash title="Terminal input"
docker run -it \
-e GX_CLOUD_ACCESS_TOKEN= YOUR_ACCESS_TOKEN \
-e GX_CLOUD_ORGANIZATION_ID= YOUR_ORGANIZATION_ID \
greatexpectations/agent:latest
```
Replace `YOUR_ACCESS_TOKEN` and `YOUR_ORGANIZATION_ID` with the values you copied previously.

3. Optional. If you created a temporary file to record your user access token and Organization ID, delete it.

4. Optional. Run the following command to use the GX Agent image as the base image and optionally add custom commands:

```bash title="Terminal input"
FROM greatexpectations/agent
RUN echo "custom_commands"
```
5. Optional. Run the following command to rebuild the Docker image:

```bash title="Terminal input"
docker build -t myorg/agent
```
6. Optional. Run `docker ps` or open Docker Desktop to confirm the agent is running.

</TabItem>
<TabItem value="kubernetes">

1. Install kubectl. See [Install Tools](https://kubernetes.io/docs/tasks/tools/).

2. Run the following command to provide the access credentials to the Kubernetes container:

```sh
kubectl create secret generic gx-agent-secret \
--from-literal=GX_CLOUD_ORGANIZATION_ID=YOUR_ORGANIZATION_ID \
--from-literal=GX_CLOUD_ACCESS_TOKEN=YOUR_ACCESS_TOKEN \
```
Replace `YOUR_ORGANIZATION_ID` and `YOUR_ACCESS_TOKEN` with the values you copied previously.

3. Optional. If you created a temporary file to record your user access token and Organization ID, delete it.

4. Create and save a file named `deployment.yaml`, with the following configuration:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: gx-agent
labels:
app: gx-agent
spec:
replicas: 1
selector:
matchLabels:
app: gx-agent
template:
metadata:
labels:
app: gx-agent
spec:
containers:
name: gx-agent
image: greatexpectations/agent:latest
envFrom:
secretRef:
name: gx-agent-secret
```
5. Run the following command to use the `deployment.yaml`configuration file to deploy the GX Agent:

```sh
kubectl apply -f deployment.yaml
```
6. Optional. Run the following command to confirm the agent is running:

```sh
kubectl logs -l app=gx-agent
```
7. Optional. Run the following command to terminate running resources gracefully:

```sh
kubectl delete -f deployment.yaml
kubectl delete secret gx-agent-secret
```


</TabItem>
</Tabs>


## Next steps

- [Create a Data Asset](../data_assets/manage_data_assets.md#create-a-data-asset)

- [Invite users](../users/manage_users.md#invite-a-user)

8 changes: 7 additions & 1 deletion docs/docusaurus/docs/cloud/connect/connect_python.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,13 @@ id: connect_python
description: Connect to a GX Cloud account and validate data from a Python script.
---

In this quickstart, you'll learn how to use GX Cloud from a Python script or interpreter, such as a Jupyter Notebook. You'll install Great Expectations, configure your GX Cloud environment variables, connect to sample data, build your first Expectation, validate data, and review the validation results through Python code.
Learn how to use GX Cloud from a Python script or interpreter, such as a Jupyter Notebook. You'll install Great Expectations, configure your GX Cloud environment variables, connect to sample data, build your first Expectation, validate data, and review the validation results through Python code.

:::tip Get the most out of GX Cloud

To get the most out of GX Cloud, GX recommends deploying the GX Agent. If you do not deploy the GX Agent, some features and functionality might be unavailable. To deploy the GX Agent, see [Deploy the GX Agent](../deploy_gx_agent.md).

:::

## Prerequisites

Expand Down
145 changes: 2 additions & 143 deletions docs/docusaurus/docs/cloud/connect/connect_snowflake.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ New to GX Cloud and not sure that it's the right solution for your organization?

- You have a [GX Cloud account](https://greatexpectations.io/cloud) with [Admin or Editor permissions](../about_gx.md#roles-and-responsibilities).

- You have deployed the GX Agent. See [Deploy the GX Agent](../deploy_gx_agent.md).

- You have a Snowflake database, schema, and table.

- You have a [Snowflake account](https://docs.snowflake.com/en/user-guide-admin) with USAGE privileges on the table, database, and schema you are validating, and you have SELECT privileges on the table you are validating. To improve data security, GX recommends using a separate Snowflake user service account to connect to GX Cloud.
Expand Down Expand Up @@ -71,151 +73,8 @@ You can use an existing Snowflake warehouse, but GX recommends creating a separa
9. Select **Run All** to allow the user with the `gx_role` role to access data on the Snowflake database and schema.
## Get your GX Cloud access token and organization ID
You'll need your access token and organization ID to set your access credentials. Don't commit your access credentials to your version control software.
1. In GX Cloud, click **Settings** > **Tokens**.
2. In the **User access tokens** pane, click **Create user access token**.
3. In the **Token name** field, enter a name for the token that will help you quickly identify it.
4. Click **Create**.
5. Copy and then paste the user access token into a temporary file. The token can't be retrieved after you close the dialog.
6. Click **Close**.
7. Copy the value in the **Organization ID** field into the temporary file with your access token and then save the file.
GX recommends deleting the temporary file after you set the environment variables.
## Deploy the GX Agent
You deploy and run the GX Agent within your deployment environment. You can deploy the GX Agent container in any environment where you can run Docker container images or create Kubernetes clusters.
To learn how to deploy a Docker container image in a specific environment, see the following documentation:
- [Quickstart: Deploy a container instance in Azure using the Azure CLI](https://learn.microsoft.com/en-us/azure/container-instances/container-instances-quickstart)
- [Build and push a Docker image with Google Cloud Build](https://cloud.google.com/build/docs/build-push-docker-image)
- [Deploy Docker Containers on Amazon ECS](https://aws.amazon.com/getting-started/hands-on/deploy-docker-containers/)
You can deploy the GX Agent in any environment in which you create Kubernetes clusters. For example:
- [Amazon Elastic Kubernetes Service (EKS)](https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html)
- [Microsoft Azure Kubernetes Service (AKS)](https://learn.microsoft.com/en-us/azure/architecture/reference-architectures/containers/aks-start-here)
- [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine/docs)
- Any Kubernetes cluster version 1.21 or greater which uses standard Kubernetes
<Tabs
groupId="connect-snowflake"
defaultValue='docker'
values={[
{label: 'Docker', value:'docker'},
{label: 'Kubernetes', value:'kubernetes'},
]}>
<TabItem value="docker">
1. Download the GX Agent Docker container image from [Docker Hub](https://hub.docker.com/r/greatexpectations/agent).
2. After configuring your cloud service to run Docker containers, run the following Docker command to start the GX Agent:
```bash title="Terminal input"
docker run -it \
-e GX_CLOUD_ACCESS_TOKEN= YOUR_ACCESS_TOKEN \
-e GX_CLOUD_ORGANIZATION_ID= YOUR_ORGANIZATION_ID \
greatexpectations/agent:latest
```
Replace `YOUR_ACCESS_TOKEN` and `YOUR_ORGANIZATION_ID` with the values you copied previously.
3. Optional. If you created a temporary file to record your user access token and Organization ID, delete it.
4. Optional. Run the following command to use the GX Agent image as the base image and optionally add custom commands:
```bash title="Terminal input"
FROM greatexpectations/agent
RUN echo "custom_commands"
```
5. Optional. Run the following command to rebuild the Docker image:
```bash title="Terminal input"
docker build -t myorg/agent
```
6. Optional. Run `docker ps` or open Docker Desktop to confirm the agent is running.
</TabItem>
<TabItem value="kubernetes">
1. Install kubectl. See [Install Tools](https://kubernetes.io/docs/tasks/tools/).
2. Run the following command to provide the access credentials to the Kubernetes container:
```sh title="Terminal input"
kubectl create secret generic gx-agent-secret \
--from-literal=GX_CLOUD_ORGANIZATION_ID=YOUR_ORGANIZATION_ID \
--from-literal=GX_CLOUD_ACCESS_TOKEN=YOUR_ACCESS_TOKEN \
```
Replace `YOUR_ORGANIZATION_ID` and `YOUR_ACCESS_TOKEN` with the values you copied previously.
3. Optional. If you created a temporary file to record your user access token and Organization ID, delete it.
4. Create and save a file named `deployment.yaml`, with the following configuration:
```yaml title="deployment.yaml"
apiVersion: apps/v1
kind: Deployment
metadata:
name: gx-agent
labels:
app: gx-agent
spec:
replicas: 1
selector:
matchLabels:
app: gx-agent
template:
metadata:
labels:
app: gx-agent
spec:
containers:
name: gx-agent
image: greatexpectations/agent:latest
envFrom:
secretRef:
name: gx-agent-secret
```
5. Run the following command to use the `deployment.yaml`configuration file to deploy the GX Agent:
```sh title="Terminal input"
kubectl apply -f deployment.yaml
```
6. Optional. Run the following command to confirm the agent is running:
```sh title="Terminal input"
kubectl logs -l app=gx-agent
```
7. Optional. Run the following command to terminate running resources gracefully:
```sh title="Terminal input"
kubectl delete -f deployment.yaml
kubectl delete secret gx-agent-secret
```
</TabItem>
</Tabs>
## Next steps
- [Create a Data Asset](../data_assets/manage_data_assets.md#create-a-data-asset)
- [Invite users](../users/manage_users.md#invite-a-user)

0 comments on commit 17d28e6

Please sign in to comment.