Consider accepting (and preferring) JSON schema for plugin 'settings' definitions

[aaronsteers]

Proposal comment: #6237 (reply in thread)

1. Allow settings to have a schema entry which would provide a JSON Schema definition for the setting.
2. If schema is provided, ignore kind - unless kind: password, and then apply the normal secret protections.
3. Similar to the SDK, we can treat a setting as secret if it has secret: true or writeOnly: true. (Either of these would cause Meltano to perform as if kind: password was set.)
4. Settings will not need sub-nodes defined, since the schema object can fully describe all child config nodes.
   • TBD: Whether to disallow child nodes from being declared in the top-level settings list, versus ignoring them, versus attempting to 'merge'.

Original discussion and background.

While comparing the SDK json format with our existing Meltano plugin files and the API / lock file format, the difference in settings definitions seemed worth calling out for further discussions.

SDK / JSON Schema

Meltano / Proprietary Schema

{ "settings": { "type": "object", "properties": { "private_token": { "type": [ "string" ], "format": "password", "description": "An access token to use when calling to the Gitlab API." }, "groups": { "type": [ "string", "null" ], "description": "A space delimited list of group ids.", "examples": [ "meltano meltanolabs" ] }, "projects": { "type": [ "string", "null" ], "description": "A space delimited list of project ids.", "examples": [ "meltano/sdk meltano/hub" ] }, "start_date": { "type": [ "string", "null" ], "format": "date-time", "description": "The start date or starting timestamp.", "examples": [ "2018-11-13T20:20:39+00:00" ] } }, "required": [ "private_token" ] } }

{ "settings": [ { "name": "access_token", "label": "Personal Access Tokens", "documentation": "https://github.com/settings/tokens", "kind": "password", "description": "Personal access token used to authenticate with GitHub." }, { "name": "repository", "label": "Repositories", "placeholder": "Ex. \"meltano/meltano meltano/sdk meltano/hub\"", "description": "Space-separated list of repositories." }, { "name": "start_date", "label": "Start Date", "kind": "date_iso8601", "description": "Defines how far into the past to pull data for the provided repositories." } ], "settings_group_validation": [ [ "access_token", "repository", "start_date" ] ] }

One path forward would be to build JSON Schema compatibility into Meltano. We could do this in a way that existing settings definitions still work but we would suggest new definitions be documented using the standard JSON Schema.

Meltano setting	JSON Schema translation
settings_group_validation	required
label	title annotation
kind:password	{"type": "string", "format": "password"}
value	default

We could detect whether the format is old-style or new-style based on whether settings is a list or an object/map.

[aaronsteers]

Coming back to this after a few months...

What if we just add a feature to allow JSON schema for plugin settings definitions?

I'll list a few options, but probably there are lots more

Essentially if a plugin has settings_jsonschema defined, it would have precedent over settings.

plugins:
  - name: my-plugin:
    settings: ... # Settings will be ignored if `config_jsonschema` is defined.
    config_jsonschema:
      # Follows is valid JSON Schema, would be grafted into a `properties` declaration:
      username:
        type: string
        description: The user name.
     password:
       type: string
       secret: True  # or writeOnly: True
       description: The user password.
     group_ids:
       type: array
       items: "string"

Meltano might do some light grooming, cleanup, or verification of the JSON Schema, but in general, we'd rely on the JSON Schema spec for it's own native expression.

Note: Above is in yaml for readability, but users/developers would also have the option to simply copy-base JSON Schema in json format.

[tayloramurphy]

@aaronsteers we actually talked about this in meltano/hub#24 (comment)

Allow both JSON schema and kind
JSON schema would override

Possibly either one of these but perhaps not both

- name: pg_password
   kind: string
   sensitive: true

#  add JSONSchema option
- name: pg_password
   kind: string # this is ignored if schema is provided
   schema: 
     type: string
     writeOnly: true

[aaronsteers]

@tayloramurphy - Thanks for re-surfacing this. 👍

And I like that proposed syntax better than what I have above. Basically settings can stay mostly unchanged but have a new schema entry to replace kind. 🚀

For sub-nodes, presumably we could then remove child node declarations from the top-level settings entry, while keeping methods like meltano config set that can inject the value at subnode levels. Does that sound right to you too?

Today users can already omit those child node declarations, but I think we'd lean towards deliberately omitting/removing them since otherwise they'd be redundant. Wdyt?

(Scroll to see the full content:)

plugins:
- name: tap-postgres
  # ...
  settings:
  - name: pg_password
    schema:
      type: string
      writeOnly: true
      secret: true
  ...
  - name: batch_config
    schema: 
      type: object
      properties:
        storage:
          type: object
          properties:
            format:
              type: string
              examples: ['jsonl']
            encoding:
              type: string
              examples: ['gzip']
  # These are then redundant with the above:
  # - name: batch_config.encoding.format
  #     type: string
  # - name: batch_config.encoding.encoding
  #     type: string

[tayloramurphy]

@aaronsteers yeah - I quite like that

[aaronsteers]

Just occurred to me, but probably worth mentioning. Our expected format will be valid JSON Schema but that doesn't necessitate that we accept any valid JSON Schema. When we get to this, we probably would want to create our own JSON Schema defining our schema expectations. (Very meta, I know. 😅 )

For instance, we may not want to have to immediately support external $ref pointers or anyOf/allOf clauses, etc. How broad our support goes would likely be influenced by how much are we using our own code to parse and validate, versus using existing libraries.

[pnadolny13]

This discussion came up in #6916 along with a few other links out to meltano/hub#24 (comment) and meltano/hub#22.

@aaronsteers I like this the proposal! We can keep using the settings definition as they are today but if theres more complex settings we can choose to implement a schema containing a jsonschema definition.