feat(compogen): use jsonref when generating the README (#99)

Because - The `compogen` tool didn't use `jsonref` to render the definition, some component readme cannot be generated. This commit - Uses `jsonref` when generating the README. - Renames `resource_specification` to `connection_specification`. - Removes the validation for an empty `connection_specification`. Note - We still do not support nested structures yet. --------- Co-authored-by: Juan Vallés <3977183+jvallesm@users.noreply.github.com> Co-authored-by: Juan Vallés <jvallesm@gmail.com>
instill-ai · Apr 24, 2024 · ff49157 · ff49157
1 parent 8ecbede
commit ff49157
Show file tree

Hide file tree

Showing 41 changed files with 1,407 additions and 329 deletions.
diff --git a/.github/workflows/test-compogen.yml b/.github/workflows/test-compogen.yml
@@ -0,0 +1,36 @@
+name: Compogen test
+
+on:
+  pull_request:
+    paths:
+      - 'tools/compogen/**'
+  push:
+    branches:
+      - main
+    paths:
+      - 'tools/compogen/**'
+
+jobs:
+  test-compogen:
+    name: test-compogen
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+        with:
+          token: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Load .env file
+        uses: cardinalby/export-env-action@v2
+        with:
+          envFile: .env
+          expand: true
+
+      - uses: actions/setup-go@v3
+        with:
+          go-version: ${{ env.GOLANG_VERSION }}
+
+      - name: Test compogen
+        run: |
+          cd tools/compogen
+          go mod tidy
+          go test -mod=mod -race -cover ./...
diff --git a/pkg/connector/archetypeai/v0/README.mdx b/pkg/connector/archetypeai/v0/README.mdx
@@ -20,53 +20,12 @@ It can carry out the following tasks:
 
 The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/connector/archetypeai/v0/config/definition.json).
 
+## Connection
+
 | Field | Field ID | Type | Note |
 | :--- | :--- | :--- | :--- |
 | API Key (required) | `api_key` | string | Fill your Archetype AI API key |
 
-Archetype AI connector resources can be created in two ways:
-
-### No-code Setup
-
-1. Go to the **Connectors** page and click **+ Create Connector**.
-1. Select **Archetype AI**.
-1. Fill in a unique ID for the resource. Optionally, give a short description in the **Description** field.
-1. Fill in the required fields described in [Resource Configuration](#resource-configuration).
-
-### Low-code Setup
-
-<CH.Code>
-
-```shellscript cURL(Instill-Cloud)
-curl -X POST https://api.instill.tech/vdp/v1beta/users/<user-id>/connectors \
---header 'Authorization: Bearer <Instill-Cloud-API-Token>' \
---data '{
-  "id": "my-archetype-ai",
-  "connector_definition_name": "connector-definitions/archetype-ai",
-  "description": "Optional description",
-  "configuration": {
-    "api_key": <api-key>
-  }
-}'
-```
-
-```shellscript cURL(Instill-Core)
-curl -X POST http://localhost:8080/vdp/v1beta/users/<user-id>/connectors \
---header 'Authorization: Bearer <Instill-Core-API-Token>' \
---data '{
-  "id": "my-archetype-ai",
-  "connector_definition_name": "connector-definitions/archetype-ai",
-  "description": "Optional description",
-  "configuration": {
-    "api_key": <api-key>
-  }
-}'
-```
-
-</CH.Code>
-
-For other component operations, please refer to the [API reference](https://openapi.instill.tech/reference/pipelinepublicservice).
-
 ## Supported Tasks
 
 ### Describe

diff --git a/pkg/connector/bigquery/v0/README.mdx b/pkg/connector/bigquery/v0/README.mdx
@@ -0,0 +1,43 @@
+---
+title: "BigQuery"
+lang: "en-US"
+draft: false
+description: "Learn about how to set up a VDP BigQuery connector https://github.com/instill-ai/instill-core"
+---
+
+The BigQuery component is a data connector that allows users to insert data to BigQuery tables.
+It can carry out the following tasks:
+
+- [Insert](#insert)
+
+## Release Stage
+
+`Alpha`
+
+## Configuration
+
+The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/connector/bigquery/v0/config/definition.json).
+
+## Connection
+
+| Field | Field ID | Type | Note |
+| :--- | :--- | :--- | :--- |
+| JSON Key File contents (required) | `json_key` | string | Contents of the JSON key file with access to the bucket. |
+| BigQuery Project ID (required) | `project_id` | string | Fill in your BigQuery Project ID. |
+| BigQuery Dataset ID (required) | `dataset_id` | string | Fill in your BigQuery Dataset ID. |
+| BigQuery Table Name (required) | `table_name` | string | Fill in your BigQuery Table Name. |
+
+## Supported Tasks
+
+### Insert
+
+Insert data to BigQuery.
+
+| Input | ID | Type | Description |
+| :--- | :--- | :--- | :--- |
+| Task ID (required) | `task` | string | `TASK_INSERT` |
+| Data (required) | `data` | object | The data to be inserted to BigQuery |
+
+| Output | ID | Type | Description |
+| :--- | :--- | :--- | :--- |
+| Status | `status` | string | Status of the upload operation |
diff --git a/pkg/connector/bigquery/v0/config/tasks.json b/pkg/connector/bigquery/v0/config/tasks.json
@@ -7,6 +7,7 @@
         "data": {
           "additionalProperties": false,
           "instillShortDescription": "The data to be inserted to BigQuery",
+          "description": "The data to be inserted to BigQuery",
           "instillUIOrder": 0,
           "patternProperties": {
             "^[a-z_][-a-z_0-9]{0,31}$": {

diff --git a/pkg/connector/bigquery/v0/main.go b/pkg/connector/bigquery/v0/main.go
@@ -1,3 +1,4 @@
+//go:generate compogen readme --connector ./config ./README.mdx
 package bigquery
 
 import (

diff --git a/pkg/connector/googlecloudstorage/v0/README.mdx b/pkg/connector/googlecloudstorage/v0/README.mdx
@@ -18,56 +18,13 @@ It can carry out the following tasks:
 
 The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/connector/googlecloudstorage/v0/config/definition.json).
 
+## Connection
+
 | Field | Field ID | Type | Note |
 | :--- | :--- | :--- | :--- |
 | Bucket Name (required) | `bucket_name` | string | Name of the bucket to be used for object storage. |
 | JSON Key File contents (required) | `json_key` | string | Contents of the JSON key file with access to the bucket. |
 
-Google Cloud Storage connector resources can be created in two ways:
-
-### No-code Setup
-
-1. Go to the **Connectors** page and click **+ Create Connector**.
-1. Select **Google Cloud Storage**.
-1. Fill in a unique ID for the resource. Optionally, give a short description in the **Description** field.
-1. Fill in the required fields described in [Resource Configuration](#resource-configuration).
-
-### Low-code Setup
-
-<CH.Code>
-
-```shellscript cURL(Instill-Cloud)
-curl -X POST https://api.instill.tech/vdp/v1beta/users/<user-id>/connectors \
---header 'Authorization: Bearer <Instill-Cloud-API-Token>' \
---data '{
-  "id": "my-gcs",
-  "connector_definition_name": "connector-definitions/gcs",
-  "description": "Optional description",
-  "configuration": {
-    "bucket_name": <bucket-name>,
-    "json_key": <json-key>
-  }
-}'
-```
-
-```shellscript cURL(Instill-Core)
-curl -X POST http://localhost:8080/vdp/v1beta/users/<user-id>/connectors \
---header 'Authorization: Bearer <Instill-Core-API-Token>' \
---data '{
-  "id": "my-gcs",
-  "connector_definition_name": "connector-definitions/gcs",
-  "description": "Optional description",
-  "configuration": {
-    "bucket_name": <bucket-name>,
-    "json_key": <json-key>
-  }
-}'
-```
-
-</CH.Code>
-
-For other component operations, please refer to the [API reference](https://openapi.instill.tech/reference/pipelinepublicservice).
-
 ## Supported Tasks
 
 ### Upload

diff --git a/pkg/connector/googlesearch/v0/README.mdx b/pkg/connector/googlesearch/v0/README.mdx
@@ -18,56 +18,13 @@ It can carry out the following tasks:
 
 The component configuration is defined and maintained [here](https://github.com/instill-ai/component/blob/main/pkg/connector/googlesearch/v0/config/definition.json).
 
+## Connection
+
 | Field | Field ID | Type | Note |
 | :--- | :--- | :--- | :--- |
 | API Key (required) | `api_key` | string | API Key for the Google Custom Search API. You can create one here: https://developers.google.com/custom-search/v1/overview#api_key |
 | Search Engine ID (required) | `cse_id` | string | ID of the Search Engine to use. Before using the Custom Search JSON API you will first need to create and configure your Programmable Search Engine. If you have not already created a Programmable Search Engine, you can start by visiting the Programmable Search Engine control panel https://programmablesearchengine.google.com/controlpanel/all. You can find this in the URL of your Search Engine. For example, if the URL of your search engine is https://cse.google.com/cse.js?cx=012345678910, the ID value is: 012345678910 |
 
-Google Search connector resources can be created in two ways:
-
-### No-code Setup
-
-1. Go to the **Connectors** page and click **+ Create Connector**.
-1. Select **Google Search**.
-1. Fill in a unique ID for the resource. Optionally, give a short description in the **Description** field.
-1. Fill in the required fields described in [Resource Configuration](#resource-configuration).
-
-### Low-code Setup
-
-<CH.Code>
-
-```shellscript cURL(Instill-Cloud)
-curl -X POST https://api.instill.tech/vdp/v1beta/users/<user-id>/connectors \
---header 'Authorization: Bearer <Instill-Cloud-API-Token>' \
---data '{
-  "id": "my-google-search",
-  "connector_definition_name": "connector-definitions/google-search",
-  "description": "Optional description",
-  "configuration": {
-    "api_key": <api-key>,
-    "cse_id": <cse-id>
-  }
-}'
-```
-
-```shellscript cURL(Instill-Core)
-curl -X POST http://localhost:8080/vdp/v1beta/users/<user-id>/connectors \
---header 'Authorization: Bearer <Instill-Core-API-Token>' \
---data '{
-  "id": "my-google-search",
-  "connector_definition_name": "connector-definitions/google-search",
-  "description": "Optional description",
-  "configuration": {
-    "api_key": <api-key>,
-    "cse_id": <cse-id>
-  }
-}'
-```
-
-</CH.Code>
-
-For other component operations, please refer to the [API reference](https://openapi.instill.tech/reference/pipelinepublicservice).
-
 ## Supported Tasks
 
 ### Search
@@ -84,4 +41,4 @@ Search data via Google Search Engine.
 
 | Output | ID | Type | Description |
 | :--- | :--- | :--- | :--- |
-| Results | `results` | array | The returned search results from Google |
+| Results | `results` | array[object] | The returned search results from Google |