From 7c6f7cbecafab043a5543f1cbef5f9beb881324b Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Fri, 6 Dec 2024 18:27:46 +0000 Subject: [PATCH 1/4] add collections CLI docs --- docs/build-for-developers/cli-collections.md | 218 +++++++++++++++++++ sidebars-main.js | 1 + 2 files changed, 219 insertions(+) create mode 100644 docs/build-for-developers/cli-collections.md diff --git a/docs/build-for-developers/cli-collections.md b/docs/build-for-developers/cli-collections.md new file mode 100644 index 000000000000..5eedfeb75b94 --- /dev/null +++ b/docs/build-for-developers/cli-collections.md @@ -0,0 +1,218 @@ +--- +title: Collections CLI Usage +sidebar_label: Collections +slug: /collections-cli +--- + +The OpenFn CLI includes support for reading from and writing to +[Collections](/documentation/build/collections) + +You can use the CLI to: + +- Explore the contents of Collections without running a Workflow +- Experiment with query syntax to get the keys you need +- Update mapping objects and lookup tables from local (or source-controlled) + files +- Manually remove unneeded data + +:::tip + +Got feedback? Want more Collections support in the CLI? Post a Feature Request +to [community.openfn.org](https://community.openfn.org/c/feature-requests)! + +::: + +Get started with the Collections API with `openfn collections --help` + +You'll need a Personal Access Token (PAT) to access a collection. You'll also +need to ensure a collection has been created before you can read or write to +it - see +[Managing Collections](http://localhost:3000/documentation/build/collections#managing-collections) + +:::info Want to use Collections in a CLI workflow? + +These docs explain how to use the `openfn collections` CLI command. + +If you're running an expression or workflow from the CLI, you need to use the +collections adaptor - check out the +[Collections Adaptor Docs](http://localhost:3000/adaptors/collections#cli-usage) +for detauls + +::: + +## Getting a PAT + +Data inside Collections is securely stored under a Project, and access is +strictly only allowed to users with access to that Project. + +So if you want to access a Collection, you have to tell the server who you are. + +We do this using Personal Access Tokens. See +[Create and Manage API Tokens](/documentation/api-tokens#about-api-tokens) for +more details. + +One you have a PAT, you need to pass it in to the CLI. The easiest way to do +this is to set your `OPENFN_PAT` env var, which the CLI will use automatically. + +If you're using multiple access tokens, you can pass `--token` to the CLI to +override the default. + +```bash +openfn collections get my-collection \* --token $MY_OPENFN_PAT +``` + +:::tip + +The rest of this guide assumes that the `OPENFN_PAT` env var has been set. So +long as it has, as you're using a server which has a `my-collection` collection, +all examples will work. + +::: + +## Fetching items + +You can fetch items from a Collection by passing a collection name and a key, or +key pattern (like `*` for "everything", or `2024*` for keys starting with +`2024`) + +```bash +openfn collections get +``` + +For example, to get everything from `my-collection`, run: + +```bash +openfn collections get my-collection \* +``` + +:::tip + +In unix shells (MacOS or Linux), the `*` character has special meaning. So if +you want to get all items, you have to escape it or quote it: + +``` +openfn collections get my-collection \* +``` + +Including \* in a pattern string should still work: + +``` +openfn collections get my-collection 2024\* +``` + +::: + +Collections are saved as strings, but will be serialized to JSON in the output. + +By default the CLI will log returned values to stdout in your shell. To write to +disk, pass `--output` or `-o` with a file path relative to your working +directory: + +```bash +openfn collections get my-collection \* -o /tmp/my_collection.json +``` + +It's important to understand that the output works a bit differently if you're +getting one item, or potentially getting many items with a pattern. + +A single always returns its value verbatim, without the key. + +So this: + +```bash +openfn collections get my-collection item-1 +``` + +Returns something like this: + +```js +{ + "id": "item-1" + /* ... other properties of the value */ +} +``` + +If you use a key-pattern to retrieve data, the value is output in multi-item +mode, which is a JSON object where the key is the item's key, and the value is +the item's value: + +``` +$ openfn collections get my-collection item-1* + +{ + "item-1": { + "id": "item-1" + /* ... other properties of the value */ + }, + "item-10": { + "id": "item-10" + /* ... other properties of the value */ + }, +} +``` + +## Uploading items + +You can use the collections command to upload data to a collection. When +uploading, values always from from a file on disk. In this example we'll use +JSON files, but if you're uploading a single value it doesn't have to be valid +JSON. + +The `set` command has two modes. To upload a single item, use: + +```bash +openfn collections set +``` + +This will read the data in path/to/value.json as a string, and upsert it under +the provided key. Key patterns are not supported. + +To bulk upsert multiple values, use: + +```bash +openfn collections set --items +``` + +The `items.json` file must contain a JSON object where the keys are item keys +and the values are item values (just like the multi-get command returns): + +```json +{ + "item-1": { + "id": "item-1" + /* ... other properties of the value */ + }, + "item-10": { + "id": "item-10" + /* ... other properties of the value */ + } +} +``` + +:::tip + +Remember that Collections always uses an _upsert_ strategy when uploading new +items. + +This means that if a key does not exist, it will be created and assigned a +value. If it already exists, its value will be updated. ::: + +::: + +## Removing items + +You can also remove items from a collection with the `collections remove` +command: + +```bash +openfn collections remove +``` + +Key-patterns are supported and allow you to remove multiple keys. + +Use `--dry-run` to get a list of the keys that would be deleted without actually +running the delete: + +```bash +openfn collections remove my-collection 2024* --dry-run +``` diff --git a/sidebars-main.js b/sidebars-main.js index 73552380fb76..91db46d24e10 100644 --- a/sidebars-main.js +++ b/sidebars-main.js @@ -122,6 +122,7 @@ module.exports = { 'build-for-developers/cli-usage', 'build-for-developers/cli-walkthrough', 'build-for-developers/cli-challenges', + 'build-for-developers/cli-collections', ], }, { From f84395d3e1bdc7234c31952a868d2083386f83d1 Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Mon, 9 Dec 2024 14:41:53 +0000 Subject: [PATCH 2/4] collectios: docs improvements --- adaptors/collections.md | 17 ++++----- docs/build-for-developers/cli-collections.md | 37 +++++++++++--------- 2 files changed, 29 insertions(+), 25 deletions(-) diff --git a/adaptors/collections.md b/adaptors/collections.md index 2ef3de8b7450..1fc2cb32e6d7 100644 --- a/adaptors/collections.md +++ b/adaptors/collections.md @@ -256,21 +256,22 @@ fn(state => { ## CLI usage -:::info +Workflows which use Collections can be run through the CLI. You will need to: -Improved Collections support is coming to the CLI soon. +- Get a Personal Access Token (PAT) +- Update the `workflow.json` with your PAT and the OpenFn endpoint +- Set the step to use the Collections adaptor + +:::tip + +You can also call the Collections API directly from the CLI. See the +[CLI Collections Guide](/documentation/collections-cli) ::: Collections are designed for close integration with the platform app, but can be used from the CLI too. -You will need to: - -- Set the job to use two adaptors -- Pass a Personal Access Token -- Set the Collections endpoint - You can get a Personal Access Token from any v2 deployment. Remember that a Collection must be created from the Admin page before it can be diff --git a/docs/build-for-developers/cli-collections.md b/docs/build-for-developers/cli-collections.md index 5eedfeb75b94..8a5451d6463a 100644 --- a/docs/build-for-developers/cli-collections.md +++ b/docs/build-for-developers/cli-collections.md @@ -43,9 +43,8 @@ for detauls ## Getting a PAT Data inside Collections is securely stored under a Project, and access is -strictly only allowed to users with access to that Project. - -So if you want to access a Collection, you have to tell the server who you are. +strictly only allowed to users with access to that Project. So if you want to +access a Collection, you have to tell the server who you are. We do this using Personal Access Tokens. See [Create and Manage API Tokens](/documentation/api-tokens#about-api-tokens) for @@ -97,16 +96,15 @@ openfn collections get my-collection \* Including \* in a pattern string should still work: ``` -openfn collections get my-collection 2024\* +openfn collections get my-collection 2024* ``` ::: Collections are saved as strings, but will be serialized to JSON in the output. -By default the CLI will log returned values to stdout in your shell. To write to -disk, pass `--output` or `-o` with a file path relative to your working -directory: +By default the CLI will log downloaded values to your shell. To write to disk, +pass `--output` or `-o` with a file path relative to your working directory: ```bash openfn collections get my-collection \* -o /tmp/my_collection.json @@ -115,15 +113,14 @@ openfn collections get my-collection \* -o /tmp/my_collection.json It's important to understand that the output works a bit differently if you're getting one item, or potentially getting many items with a pattern. -A single always returns its value verbatim, without the key. - -So this: +A single key always returns its value "raw" or "verbatim", without the key +attached. So for a key `item-1` which holds a JSON object as a value, then this: ```bash openfn collections get my-collection item-1 ``` -Returns something like this: +Will download and save something like this: ```js { @@ -133,12 +130,18 @@ Returns something like this: ``` If you use a key-pattern to retrieve data, the value is output in multi-item -mode, which is a JSON object where the key is the item's key, and the value is +mode: which is a JSON object where the key is the item's key, and the value is the item's value: -``` +So if we get all items whose key starts with `item-`: + +```bash $ openfn collections get my-collection item-1* +``` + +The resulting data will look like this: +```json { "item-1": { "id": "item-1" @@ -147,14 +150,14 @@ $ openfn collections get my-collection item-1* "item-10": { "id": "item-10" /* ... other properties of the value */ - }, + } } ``` ## Uploading items You can use the collections command to upload data to a collection. When -uploading, values always from from a file on disk. In this example we'll use +uploading, values always come from a file on disk. In this example we'll use JSON files, but if you're uploading a single value it doesn't have to be valid JSON. @@ -164,7 +167,7 @@ The `set` command has two modes. To upload a single item, use: openfn collections set ``` -This will read the data in path/to/value.json as a string, and upsert it under +This will read the data in `path/to/value.json` as a string, and upsert it under the provided key. Key patterns are not supported. To bulk upsert multiple values, use: @@ -174,7 +177,7 @@ openfn collections set --items ``` The `items.json` file must contain a JSON object where the keys are item keys -and the values are item values (just like the multi-get command returns): +and the values are item values (just like the multi-item get command returns): ```json { From 3d9d5e14080ea468dc75adc5329001c795857ddc Mon Sep 17 00:00:00 2001 From: Joe Clark Date: Thu, 12 Dec 2024 14:37:48 +0000 Subject: [PATCH 3/4] updates --- docs/build-for-developers/cli-collections.md | 53 ++++++++++++++++++-- 1 file changed, 48 insertions(+), 5 deletions(-) diff --git a/docs/build-for-developers/cli-collections.md b/docs/build-for-developers/cli-collections.md index 8a5451d6463a..2c6e3ee30f02 100644 --- a/docs/build-for-developers/cli-collections.md +++ b/docs/build-for-developers/cli-collections.md @@ -5,7 +5,16 @@ slug: /collections-cli --- The OpenFn CLI includes support for reading from and writing to -[Collections](/documentation/build/collections) +[Collections](/documentation/build/collections): a key/value store built into +OpenFn. + +:::caution Versions + +Collections support was added to the CLI in version 1.9.0. + +Run `npm install -g @openfn/cli` to update or install. + +::: You can use the CLI to: @@ -29,11 +38,11 @@ need to ensure a collection has been created before you can read or write to it - see [Managing Collections](http://localhost:3000/documentation/build/collections#managing-collections) -:::info Want to use Collections in a CLI workflow? +:::info Trying to use Collections in a CLI workflow? These docs explain how to use the `openfn collections` CLI command. -If you're running an expression or workflow from the CLI, you need to use the +If you're running an expression or workflow through the CLI, you need to use the collections adaptor - check out the [Collections Adaptor Docs](http://localhost:3000/adaptors/collections#cli-usage) for detauls @@ -68,6 +77,33 @@ all examples will work. ::: +## Setting a server + +By default, the CLI will all out to our primary platform at +https://app.openfn.org. + +If you're running from open source or using a different deployment, you'll also +need to tell the CLI which Collections server to use. + +You can do this by passing `--lightning` directly: + +```bash +openfn collections get my-collection \* --lightning http://localhost:4000 +``` + +Or by setting the `OPENFN_ENDPOINT` environment variable. + +:::tip + +To see which server the CLI is using, ask for debug-level logging in your +output: + +```bash +openfn collections get my-collection \* --log debug +``` + +::: + ## Fetching items You can fetch items from a Collection by passing a collection name and a key, or @@ -110,8 +146,15 @@ pass `--output` or `-o` with a file path relative to your working directory: openfn collections get my-collection \* -o /tmp/my_collection.json ``` +To format the output to make it easier to read, add the `--pretty` flag for +pretty-printing: + +```bash +openfn collections get my-collection \* -o /tmp/my_collection.json --pretty +``` + It's important to understand that the output works a bit differently if you're -getting one item, or potentially getting many items with a pattern. +getting one specific item with or getting many items with a key-pattern. A single key always returns its value "raw" or "verbatim", without the key attached. So for a key `item-1` which holds a JSON object as a value, then this: @@ -198,7 +241,7 @@ Remember that Collections always uses an _upsert_ strategy when uploading new items. This means that if a key does not exist, it will be created and assigned a -value. If it already exists, its value will be updated. ::: +value. If it already exists, its value will be updated. ::: From f0c321c25fae8f119070b73219ff8630f1a5c88d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Rita=20Z=C3=A1goni?= Date: Thu, 12 Dec 2024 15:44:44 +0100 Subject: [PATCH 4/4] Update cli-collections.md --- docs/build-for-developers/cli-collections.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/build-for-developers/cli-collections.md b/docs/build-for-developers/cli-collections.md index 2c6e3ee30f02..8f6a13b3e598 100644 --- a/docs/build-for-developers/cli-collections.md +++ b/docs/build-for-developers/cli-collections.md @@ -199,7 +199,7 @@ The resulting data will look like this: ## Uploading items -You can use the collections command to upload data to a collection. When +You can use the `collections` command to upload data to a collection. When uploading, values always come from a file on disk. In this example we'll use JSON files, but if you're uploading a single value it doesn't have to be valid JSON.