[EPIC] Add title and description properties to each field in connectors specifications and order input fields appropriately

[sherifnada]

Context

Each connector declares a specification, a json object which describes the inputs that this connector can take as input in order to run a sync. For example:

{
  "type": "object",
  "properties": {
    "user_name": {
      "type": "string"
    }
  }
}

indicates that a connector takes in a username string in order to run.

Additionally, the description of each property can be enriched with the title and description attributes to produce a more user friendly experience in the UI. This displays this information to the user in a polished way and gives less technical users (e.g: analysts) confidence that they can use this product. This is especially important because a non technical user will lose confidence in the product if they see something like user_name instead of Username because they will feel it's unpolished and perhaps not ready for production usage. For example the following spec:

{
  "type": "object",
  "properties": {
    "user_name": {
      "type": "string"
    }
  }
}

produces the following input field in the UI:

Whereas the following specification:

{
  "type": "object",
  "properties": {
    "user_name": {
      "type": "string",
      "description": "A username with read access and create schema permissions in your database.",
      "title": "Username"
    }
  }
}

Tell us about the problem you're trying to solve

We want to provide title and description properties for each field for each connector in order to provide a good UX for users.

Conventions for how to write good title and description fields

title: This should be a "human readable" form of the connector's parameter name. For example, if the parameter is called account_id the title should be Account ID. However, the title property doesn't have to be the same as the the parameter name. For example, if the property name is confusing or difficult to understand e.g it's called key, this is a very unintuitive name to the user; in this case consider renaming it to something like API key.

description: Describes what the parameter is for. Be specific and unambiguous in the wording, explaining more than just the field name alone provides. Specifically, you should explain the following:

1. If the field is optional, what happens if I don't set the field? For example, if a connector has an optional Start Date parameter, a good description might be Any data created after this date will be replicated. If this date is not set, the connector replicates data from 1 year ago by default.
2. What are the limits or assumptions associated with this parameter? for example, if it's API Token, what permission level should the token have? (usually it's something like An API token with read permissions to all your Shopify resources).
3. If it's a date, what is the format of the input? e.g: YYYY-MM-DD or YYYY-MM-DDT00:00:00Z etc.
4. Should you add a link to any documentation? You can do this using the HTML <a href="URL_HERE">NAME_OF_THE_LINK</a>.
5. Add any other information that helps the user understand how to input this field correctly

To do this well, you may need to dig into what the meaning of the parameter is. For example, if you see a parameter like conversion_window_days and you're not sure what that means, you'll need to either ask the eng team for help, look at the connector code or API documentation to understand what it is.

Implementation

There are two ways of composing specifications:

1. JSON-based i.e: it's just a JSON file which is displayed exactly as it is without any alteration
2. code-based like this one for example. In this case, this is converted to JSON first (and may sometimes have implicit fields e.g: automatically adds the ) before being output from the connector.

In both cases however, the final spec is always rendered as JSON. To see the connector's finally rendered spec run docker run <connector_name> spec e.g: docker run airbyte/source-postgres spec. If you are running windows, you may need to install WSL2 and install docker.

Once you have the final JSON specification, you can always preview what it will look like in the UI using instructions here.

The process you should follow is:

1. Run the connector's spec method using the docker command above to find out the final spec currently rendered by the connector
2. See which fields do not have title/description properties
3. Update them accordingly.
4. Create a PR and ask for review from Airbyte on the changes. It's probably much easier to batch changes in bigger PRs (let's say 5 or 10 connectors per PR) rather than create one PR per connector.
5. publish the connector while skipping tests. This way it doesn't matter if the build is failing or not. However, be super careful not to change anything about the connector other than adding title and description -- this is safe to do only if title and description properties were the only thing that was changed. To publish a connector while skipping tests add the run-tests=false clause to the end of the /publish command e.g: /publish connector=connectors/source-postgres run-tests=false.

At times, the build will fail if you format the JSON incorrectly. Don't worry about it, just tell the eng team and they'll fix it. Or if you're feeling adventurous ask them to show you how to run the formatting commands using gradle so you can fix the build yourself.

Challenges you'll need engineering help with

Ask the engineering team for assistance if:

1. you're not sure how to verify the spec output by an actual connector (they may need to help you run a command like the docker command above)
2. you're not sure what the field does and therefore not sure what the title or description of the field should be
3. Any other technical blocker is happening

[igrankova]

https://docs.google.com/spreadsheets/d/1SS_fxBOilj2z0uPNKEwDFnOpLdsbH18x3YwwLYyJZWw/edit?usp=sharing

Standardize config labels on source/destination setup #7841

[igrankova]

@sherifnada, We have several questions to specify updates details:

1. For instance we have the field title "Start Date" and we received a requirement in the task 7841, that it should be written with uppercase both words. So we should update the other fields like "Source type" to "Source Type" as well, right? But what if our field title is "Requests per minute"? Should we update it to "Request per Minute" for example or leave as it is? And I saw the other longer fields titles as well.
2. The link to documentation - how should we decide that the link is needed? What criteria tells us, that we need this link or we should add links to docs for all connectors?