diff --git a/adaptors/collections.md b/adaptors/collections.md index 2ef3de8b7450..1fc2cb32e6d7 100644 --- a/adaptors/collections.md +++ b/adaptors/collections.md @@ -256,21 +256,22 @@ fn(state => { ## CLI usage -:::info +Workflows which use Collections can be run through the CLI. You will need to: -Improved Collections support is coming to the CLI soon. +- Get a Personal Access Token (PAT) +- Update the `workflow.json` with your PAT and the OpenFn endpoint +- Set the step to use the Collections adaptor + +:::tip + +You can also call the Collections API directly from the CLI. See the +[CLI Collections Guide](/documentation/collections-cli) ::: Collections are designed for close integration with the platform app, but can be used from the CLI too. -You will need to: - -- Set the job to use two adaptors -- Pass a Personal Access Token -- Set the Collections endpoint - You can get a Personal Access Token from any v2 deployment. Remember that a Collection must be created from the Admin page before it can be diff --git a/docs/build-for-developers/cli-collections.md b/docs/build-for-developers/cli-collections.md new file mode 100644 index 000000000000..8f6a13b3e598 --- /dev/null +++ b/docs/build-for-developers/cli-collections.md @@ -0,0 +1,264 @@ +--- +title: Collections CLI Usage +sidebar_label: Collections +slug: /collections-cli +--- + +The OpenFn CLI includes support for reading from and writing to +[Collections](/documentation/build/collections): a key/value store built into +OpenFn. + +:::caution Versions + +Collections support was added to the CLI in version 1.9.0. + +Run `npm install -g @openfn/cli` to update or install. + +::: + +You can use the CLI to: + +- Explore the contents of Collections without running a Workflow +- Experiment with query syntax to get the keys you need +- Update mapping objects and lookup tables from local (or source-controlled) + files +- Manually remove unneeded data + +:::tip + +Got feedback? Want more Collections support in the CLI? Post a Feature Request +to [community.openfn.org](https://community.openfn.org/c/feature-requests)! + +::: + +Get started with the Collections API with `openfn collections --help` + +You'll need a Personal Access Token (PAT) to access a collection. You'll also +need to ensure a collection has been created before you can read or write to +it - see +[Managing Collections](http://localhost:3000/documentation/build/collections#managing-collections) + +:::info Trying to use Collections in a CLI workflow? + +These docs explain how to use the `openfn collections` CLI command. + +If you're running an expression or workflow through the CLI, you need to use the +collections adaptor - check out the +[Collections Adaptor Docs](http://localhost:3000/adaptors/collections#cli-usage) +for detauls + +::: + +## Getting a PAT + +Data inside Collections is securely stored under a Project, and access is +strictly only allowed to users with access to that Project. So if you want to +access a Collection, you have to tell the server who you are. + +We do this using Personal Access Tokens. See +[Create and Manage API Tokens](/documentation/api-tokens#about-api-tokens) for +more details. + +One you have a PAT, you need to pass it in to the CLI. The easiest way to do +this is to set your `OPENFN_PAT` env var, which the CLI will use automatically. + +If you're using multiple access tokens, you can pass `--token` to the CLI to +override the default. + +```bash +openfn collections get my-collection \* --token $MY_OPENFN_PAT +``` + +:::tip + +The rest of this guide assumes that the `OPENFN_PAT` env var has been set. So +long as it has, as you're using a server which has a `my-collection` collection, +all examples will work. + +::: + +## Setting a server + +By default, the CLI will all out to our primary platform at +https://app.openfn.org. + +If you're running from open source or using a different deployment, you'll also +need to tell the CLI which Collections server to use. + +You can do this by passing `--lightning` directly: + +```bash +openfn collections get my-collection \* --lightning http://localhost:4000 +``` + +Or by setting the `OPENFN_ENDPOINT` environment variable. + +:::tip + +To see which server the CLI is using, ask for debug-level logging in your +output: + +```bash +openfn collections get my-collection \* --log debug +``` + +::: + +## Fetching items + +You can fetch items from a Collection by passing a collection name and a key, or +key pattern (like `*` for "everything", or `2024*` for keys starting with +`2024`) + +```bash +openfn collections get +``` + +For example, to get everything from `my-collection`, run: + +```bash +openfn collections get my-collection \* +``` + +:::tip + +In unix shells (MacOS or Linux), the `*` character has special meaning. So if +you want to get all items, you have to escape it or quote it: + +``` +openfn collections get my-collection \* +``` + +Including \* in a pattern string should still work: + +``` +openfn collections get my-collection 2024* +``` + +::: + +Collections are saved as strings, but will be serialized to JSON in the output. + +By default the CLI will log downloaded values to your shell. To write to disk, +pass `--output` or `-o` with a file path relative to your working directory: + +```bash +openfn collections get my-collection \* -o /tmp/my_collection.json +``` + +To format the output to make it easier to read, add the `--pretty` flag for +pretty-printing: + +```bash +openfn collections get my-collection \* -o /tmp/my_collection.json --pretty +``` + +It's important to understand that the output works a bit differently if you're +getting one specific item with or getting many items with a key-pattern. + +A single key always returns its value "raw" or "verbatim", without the key +attached. So for a key `item-1` which holds a JSON object as a value, then this: + +```bash +openfn collections get my-collection item-1 +``` + +Will download and save something like this: + +```js +{ + "id": "item-1" + /* ... other properties of the value */ +} +``` + +If you use a key-pattern to retrieve data, the value is output in multi-item +mode: which is a JSON object where the key is the item's key, and the value is +the item's value: + +So if we get all items whose key starts with `item-`: + +```bash +$ openfn collections get my-collection item-1* +``` + +The resulting data will look like this: + +```json +{ + "item-1": { + "id": "item-1" + /* ... other properties of the value */ + }, + "item-10": { + "id": "item-10" + /* ... other properties of the value */ + } +} +``` + +## Uploading items + +You can use the `collections` command to upload data to a collection. When +uploading, values always come from a file on disk. In this example we'll use +JSON files, but if you're uploading a single value it doesn't have to be valid +JSON. + +The `set` command has two modes. To upload a single item, use: + +```bash +openfn collections set +``` + +This will read the data in `path/to/value.json` as a string, and upsert it under +the provided key. Key patterns are not supported. + +To bulk upsert multiple values, use: + +```bash +openfn collections set --items +``` + +The `items.json` file must contain a JSON object where the keys are item keys +and the values are item values (just like the multi-item get command returns): + +```json +{ + "item-1": { + "id": "item-1" + /* ... other properties of the value */ + }, + "item-10": { + "id": "item-10" + /* ... other properties of the value */ + } +} +``` + +:::tip + +Remember that Collections always uses an _upsert_ strategy when uploading new +items. + +This means that if a key does not exist, it will be created and assigned a +value. If it already exists, its value will be updated. + +::: + +## Removing items + +You can also remove items from a collection with the `collections remove` +command: + +```bash +openfn collections remove +``` + +Key-patterns are supported and allow you to remove multiple keys. + +Use `--dry-run` to get a list of the keys that would be deleted without actually +running the delete: + +```bash +openfn collections remove my-collection 2024* --dry-run +``` diff --git a/sidebars-main.js b/sidebars-main.js index 73552380fb76..91db46d24e10 100644 --- a/sidebars-main.js +++ b/sidebars-main.js @@ -122,6 +122,7 @@ module.exports = { 'build-for-developers/cli-usage', 'build-for-developers/cli-walkthrough', 'build-for-developers/cli-challenges', + 'build-for-developers/cli-collections', ], }, {