[RFC, v3] Switching intermediate output from Markdown to YAML (aka Project Camel)

[shalvah]

This is an RFC for a feature I plan to add in the next major version of Scribe. I'll leave it up for a while to gather any thoughts before I decide on the implementation.

What is this?

Project Camel is a move to switch Scribe's processing from [extract -> Markdown -> HTML (via Pastel)] to [extract -> YAML -> HTML]. In this model, Scribe will extract API and endpoints information and write that to a set of YAML files in a structured format, then it will transform these files into the usual HTML/Postman/OpenAPI outputs.

Why?

The main goal is to not limit you. Scribe wants to help you generate great docs, and part of that is acknowledging that it can't do everything, and making it easier for you to change/add things the package misses. The structured format allows for greater flexibility (see the section on "Benefits" below).

Implementation

When you run scribe:generate

1. If there are no existing data files (YAML files in the specified format):
   a. Scribe will extract API and endpoints information and hold in memory.
   b. Scribe will write the data to new data files and "compare files" (identical files used to compare on a future run to know what changed).
   c. Scribe will transform the data it has in memory (not the data files) into the HTML/Postman/OpenAPI outputs. (It uses data it has rather than loading the data files as a speed optimisation.) END.
2. Otherwise, if there are existing data files:
   a. If --force is set: Scribe will extract data, overwrite the data files and compare files, and continue to the usual outputs. END.
   b. If --force is not set:
   • Scribe will load the data from the data files. Scribe will also load data from the compare files separately. ("Load" means an actual Yaml::parse() into PHP arrays, not just reading the file contents.)
   • For each endpoint from the loaded data, Scribe will compare it with the corresponding data from the compare files.
     • If both are the same, Scribe will extract data for the endpoint, then overwrite the loaded data for that endpoint with the new, extracted data.
     • If there's a difference (or no data in the compare file for that endpoint), Scribe assumes a change was made by the user, and keeps that data. Scribe will not attempt to extract any data for that endpoint.
   • When Scribe is done, it writes the endpoints back to data files + compare files and continues to the usual outputs. END.

With --no-extraction:

--no-extraction requires the data files to be present. Scribe will load the data files and transform them to Markdown.

Note that the data files are loaded before route extraction is done. This allows us to skip extracting data for any routes that have been modified.

Here's what a group file could look like.

# file name could be 1.yaml
- name: User management
  description: |
    Group description. Supports Markdown.
  endpoints:
  - path: '/users/{id}',
    methods: ['GET']
    metadata: 
      title: Get a user's details
      description: # Supports markdown
      authenticated: true
    headers: 
      Content-Type: 'application/json' 
    urlParameters:
      id: 
        name: '',
        description: ''
        required: false,
        value: 3,
        type: 'integer',
    queryParameters: {}
    bodyParameters: {}
    responses:
    - status: 200
      description: ''
      content: ''
    responseFields: []

Benefits

• Easier to add other endpoints (See How to describe endpoints provided by vendor package? #21): Currently you can add another endpoint manually, but that would mean editing the Markdown files (and Postman/OpenAPI as well). With Camel, each endpoint is in a structured format, so you can just copy one and replace it with the data you want, and it will be reflected in Postman/OpenAPI when you generate. Also, this would allow you to add the endpoint to an existing group.
• Easier to modify endpoint data: Similar to above, but with an even bigger benefit: Right now, if you modify one endpoint's data, you have to forgo automatic generation for all endpoints in the group. With this, you wouldn't.
• Custom sorting (See Feature Request: Sort endpoints within groups alphabetically (by title) #99): Endpoint group files are included in alphabetical order from the groups/ directory.. To order endpoints within a group, just edit the YAML file for the group and rearrange them. Scribe will respect that order on the next run (except you `--force). Groups are written to numbered files, and you can re-number them to sort the groups.
• Easier output customisation (See Customize CSS in Documentation when 'type' => 'static'  #49, [Feature request] Config item in scribe.php for CSS #51): Since Scribe handles the direct YAML->HTML transformation, it would then be easier for us to implement customisation of the HTML. For instance, you could specify in your scribe.php, output.css.include to include extra styles, output.head.append to add scripts at the end of the HEAD tag, etc.
• Easier to implement different templates(See How do i customize colors or put a light theme? #41): Like above, but with the added detail that a templates may require the HTML to be written in a different format. Since we're coming from YAML, all we need is Blade files for the template, and corresponding CSS.

Cons

• Lack of Markdown output means any place where users might write Markdown (intro_text, endpoint descriptions) has to be manually passed through a Markdown-to-HTML parser.
• Might be slower, since for existing data files, Scribe will read the content and the compare file's content, manually compare data for each endpoint and manually update the content. On the other hand, we would gain some (minor) speed savings by cutting out the Markdown part of things.
• [Major] This approach won't work well when you change a group name in your code. The old group file from the last run would still be present in the directory and get included in the docs. The existing Markdown approach always deletes any Markdown files that were not created in this run in the groups/ directory. But we want the users to be able to add extra groups if they wish. Maybe we'll keep that approach. The user shouldn't add any new group files. Instead, they can add a new group into any existing group file. Solution: Before writing data files, Scribe will read the last known groups from the compare files and delete any that can't be found in the data files. Users add custom groups in .custom.yaml files within the data dir.

Considerations

This makes things too complicated.

Not really. It's just as complicated as the current setup, only different. Also, the aim is to hide as much complexity as possible until the user needs it. For instance, unless you want to change something in the endpoints' data or add an endpoint, you shouldn't need to touch the YAML files at all.

Why not write to (and read from) a standard format such as OpenAPI?

OpenAPI is a spec that describes how APIs should be written, not necessarily how they are. For instance, it does not support optional path parameters. Also, parsing OpenAPI into the proper route information is complicated, because of its strict format. Finally, you can't store examples and other presentation info in OpenAPI. 😕

Why YAML?

It's the most readable out there, and I want users to not feel intimidated when they dive in to these files. It's got some hiccups writing-wise, but they should be able to handle that.

[shalvah]

BTW, if anyone wants to check out the existing work on v2, here you go: https://scribe.readthedocs.io/en/v2/migrating-v2.html. It's not officially released yet, but you can try it with the dev-v2 constraint. Implementing this won't affect the things I've already implemented.

[robtesch]

Hi @shalvah look I personally dislike YAML, but this really sounds like an overall win and potentially solves a lot of the minor niggles I have had so far. I really appreciate all the work you are putting into this lib.

[shalvah]

I've decided to spend some time to develop this for v3. I'll release v2 in the next week or so, and begin work on v3.

[shalvah]

🎉v3 is out now, with all the improvements of Project Camel!

[robtesch]

Amazing!