Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhancement: Allow frontmatter.json to be split in multiple files #412

Closed
landure opened this issue Sep 16, 2022 · 25 comments
Closed

Enhancement: Allow frontmatter.json to be split in multiple files #412

landure opened this issue Sep 16, 2022 · 25 comments
Labels
enhancement New feature or request Project: v8.2.0

Comments

@landure
Copy link

landure commented Sep 16, 2022

When describing complex content types, frontmatter.json quickly become a long and unreadable file.
I just finished creating a content type describing Hugo various front-matter fields, and my frontmatter.json has 211 lines of content, and this is just with one complex content type and some frontmatter settings.

Hugo allows to split is configuration file in multiple files stored in config folder, and named according to the config section they store.
An example of Hugo splitted config can be found at https://github.com/razonyang/hugo-theme-bootstrap-skeleton/tree/main/config/_default

Frontmatter could allow a similar setup in .frontmatter/config to allow an atomic configuration.

Another option is to allow the use of an import directive in frontmatter.json where the user explicitly import another file in lieu of a parameter value.

@landure landure added the enhancement New feature or request label Sep 16, 2022
@estruyf
Copy link
Owner

estruyf commented Sep 20, 2022

This would be an interesting approach. It also needs to be carefully thought through with the #407 feature.

@michaeltlombardi
Copy link

I like this idea, this seems very similar to extending a local configuration proposal in #407 - maybe that would solve both?

I do think the DevX for being able to automatically import and build the configs in .frontmatter/config would be very useful even without a way to publish/consume configs not kept (and checked in with) the current project.

@estruyf
Copy link
Owner

estruyf commented Sep 22, 2022

@michaeltlombardi there is room for both, as I don't think this will work for your scenario.

@landure let us start creating a list of configuration/settings we want to split up in separate files. That will help to get the development for this started.

@landure
Copy link
Author

landure commented Sep 23, 2022

@estruyf From the top of my head, I see at least three configuration settings that would gain to be stored in separate files:

  • frontMatter.taxonomy.contentTypes array contents, that could be stored for example in .frontmatter/config/taxonomy.contentTypes.json or .frontmatter/config/taxonomy.contentTypes/*.json
  • frontMatter.content.snippets array contents, that could be stored for example in .frontmatter/config/content.snippets.json or .frontmatter/config/content.snippets/*.json
  • frontMatter.content.placeholders array contents, that could be stored for example in .frontmatter/config/content.placeholders.json or .frontmatter/config/content.placeholders/*.json

The idea being to be able to easily import reproducible configuration settings from elsewhere, and to make the configuration more readable and maintainable.

@michaeltlombardi
Copy link

My only suggestion to the proposal outlined is that the convention always replace . with folder level, so instead of (not reordered for alphabetic sorting, sorry):

.
├── .frontmatter/
│   └── config/
│       ├── taxonomy.contentTypes/
│       │   └── *.json
│       ├── taxonomy.contentTypes.json
│       ├── content.placeholders/
│       │   └── *.json
│       ├── content.placeholders.json
│       ├── content.snippets/
│       │   └── *.json
│       └── content.snippets.json
└── .frontmatter.json

It would be:

.
├── .frontmatter/
│   └── config/
│       └── taxonomy/
│       │   ├── contentTypes/
│       │   │   └── *.json
│       │   ├── contentTypes.json
│       └── content/
│           ├── placeholders/
│           │   └── *.json
│           ├── placeholders.json
│           ├── snippets/
│           │   └── *.json
│           └── snippets.json
└── .frontmatter.json

I think those three sections - taxonomy.contentTypes, content.snippets, and content.placeholders make enormous sense for a first pass. I think in the future it would absolutely be useful to be able to do data.*, taxonomy.fieldGroups, and taxonomy.customTaxonomy.

For me, the ideal end state is to be able to have most or all fields definable via this split definition, with non-arrays defined in their nearest ancestor .json. For example:

// .frontmatter/config/taxonomy.json
{
  "frontMatter.taxonomy.seoContentLength": 1760,
  "frontMatter.taxonomy.seoDescriptionField": "description",
  // etc for the non-arrays
}
// .frontmatter/config/taxonomy/contentTypes/foo.json
{
  "name": "foo",
  "pageBundle": false,
  "fields": [
    {
      "title": "Title",
      "name": "title",
      "type": "string",
      "single": true
    },
    // snipped for brevity
  ]
}
// .frontmatter/config/taxonomy/contentTypes/bar.json
{
  "name": "bar",
  "pageBundle": false,
  "fields": [
    {
      "title": "Title",
      "name": "title",
      "type": "string",
      "single": true
    },
    // snipped for brevity
  ]
}

With the deterministic file/folder name, we could know what schema to apply where.

On possible future DevX

In the example above, I used the full key names in the split config. To be very clear, I believe that's how this should be implemented initially because I expect it's the least amount of work. However, I think a possible enhanced UX would be this instead:

// .frontmatter/config/taxonomy.json
{
  "seoContentLength": 1760,
  "seoDescriptionField": "description",
  // etc for the non-arrays
}

But that would require some schema rework (or copying the schema partially and maintaining it separately, which seems rough - def more thought needs to go into the maintainability of something like this). I just wanted to note it here because if I didn't do it now, I would entirely forget.

I really, really like this proposal. I also think sorting this out is something of a prerequisite for #407, especially since this model would make maintaining "packaged" configurations easier and will already require some work on merging. This is also much more relevant to the majority of the user base.

My own config file is 767 lines long without the hundreds of lines I would need for my data type definitions.

@estruyf
Copy link
Owner

estruyf commented Sep 29, 2022

I started to work on how @michaeltlombardi defined it.

In the Front Matter: Diagnostic logging you'll be able to see the merged config, this will be useful to test things out, or see what the actual config is that gets applied.

Screenshot 2022-09-29 at 19 24 16

@estruyf
Copy link
Owner

estruyf commented Sep 29, 2022

The example config folders and files:

Screenshot 2022-09-29 at 20 08 09

The outcome:

{
  "$schema": "https://beta.frontmatter.codes/frontmatter.schema.json",
  "frontMatter.framework.id": "docusaurus",
  "frontMatter.content.publicFolder": "static",
  "frontMatter.taxonomy.tags": [
    "docusaurus",
    "facebook",
    "hello",
    "hola"
  ],
  "frontMatter.taxonomy.categories": [],
  "frontMatter.content.pageFolders": [
    {
      "title": "blog",
      "path": "[[workspace]]/blog"
    },
    {
      "title": "docs",
      "path": "[[workspace]]/docs"
    }
  ],
  "frontMatter.content.placeholders": [
    {
      "id": "ogImage",
      "script": "./scripts/og-image.js",
      "command": "~/.nvm/versions/node/v16.11.1/bin/node"
    }
  ],
  "frontMatter.content.snippets": {
    "blockquote": {
      "body": "{{< blockquote type=\"[[type]]\" text=\"[[&selection]]\" >}}",
      "description": "Creates a blockquote",
      "fields": [
        {
          "name": "type",
          "title": "Type",
          "type": "choice",
          "choices": [
            "info",
            "important"
          ],
          "default": "info"
        },
        {
          "name": "selection",
          "title": "Selection",
          "type": "string",
          "default": "FM_SELECTED_TEXT"
        }
      ]
    },
    "highlight": {
      "description": "Creates a code highlighting box",
      "body": [
        "{{< highlight \"[[type]]\" \"linenos=table,noclasses=false\" >}}",
        "  [[selection]]",
        "{{< / highlight >}}"
      ],
      "fields": [
        {
          "name": "type",
          "title": "Language",
          "type": "choice",
          "choices": [
            "html",
            "css",
            "typescript"
          ],
          "default": "typescript"
        },
        {
          "name": "selection",
          "title": "Selection",
          "type": "string",
          "default": "FM_SELECTED_TEXT"
        }
      ]
    },
    "image-snippet": {
      "body": "{{< caption-new \"[[&mediaUrl]]\" \"[[caption]]\" \"[[customCaption]]\" >}}",
      "isMediaSnippet": true,
      "description": "",
      "fields": [
        {
          "name": "customCaption",
          "title": "Custom caption",
          "type": "string",
          "default": "FM_SELECTED_TEXT"
        }
      ]
    },
    "video-snippet": {
      "body": [
        "{{< video \"[[&mediaUrl]]\" \"[[caption]]\" >}}"
      ],
      "isMediaSnippet": true
    }
  },
  "frontMatter.taxonomy.contentTypes": [
    {
      "name": "blog",
      "pageBundle": false,
      "previewPath": null,
      "fields": [
        {
          "title": "Title",
          "name": "title",
          "type": "string"
        },
        {
          "title": "Description",
          "name": "description",
          "type": "string"
        }
      ]
    },
    {
      "name": "default",
      "pageBundle": false,
      "previewPath": null,
      "fields": [
        {
          "title": "Title",
          "name": "title",
          "type": "string"
        },
        {
          "title": "Description",
          "name": "description",
          "type": "string"
        },
        {
          "title": "Publishing date",
          "name": "date",
          "type": "datetime",
          "default": "{{now}}",
          "isPublishDate": true
        },
        {
          "title": "Content preview",
          "name": "preview",
          "type": "image"
        },
        {
          "title": "Is in draft",
          "name": "draft",
          "type": "draft"
        },
        {
          "title": "Tags",
          "name": "tags",
          "type": "tags"
        },
        {
          "title": "Categories",
          "name": "categories",
          "type": "categories"
        }
      ]
    },
    {
      "name": "post",
      "pageBundle": false,
      "previewPath": null,
      "fields": [
        {
          "title": "Title",
          "name": "title",
          "type": "string"
        },
        {
          "title": "Description",
          "name": "description",
          "type": "string"
        },
        {
          "title": "Tags",
          "name": "tags",
          "type": "tags"
        }
      ]
    }
  ]
}

@estruyf
Copy link
Owner

estruyf commented Sep 29, 2022

For the snippets, it would be useful to have a new title property, as the filename is currently used as the key. An optional title property will allow overriding of the title shown on the dashboard.

This is how it gets rendered at the moment:

image

estruyf added a commit that referenced this issue Sep 29, 2022
@estruyf
Copy link
Owner

estruyf commented Sep 29, 2022

Pushed the current change for the new beta. The feature is not yet complete, but already available to give it a try to see if we're on the right path.

Current settings which can be split into folders/files are:

  • Content-types: .frontmatter/config/taxonomy/contentTypes
  • Page folders: .frontmatter/config/taxonomy/pageFolders
  • Placeholders: .frontmatter/config/taxonomy/placeholders
  • Snippets: .frontmatter/config/taxonomy/snippets

I've also created a sample project: https://github.com/FrontMatter/project-samples/tree/main/docusaurus

Let me know what you think.

@estruyf
Copy link
Owner

estruyf commented Sep 30, 2022

Testing out if we can split the JSON schema into multiple files. That will make it easier to reuse these in the config files. For instance, this is what the content-type schema looks like: https://beta.frontmatter.codes/config/taxonomy.contenttype.schema.json

{
  "$schema": "https://beta.frontmatter.codes/config/taxonomy.contenttype.schema.json",
  "name": "blog",
  "pageBundle": false,
  "previewPath": null,
  "fields": [
    {
      "title": "Title",
      "name": "title",
      "type": "string"
    },
    {
      "title": "Description",
      "name": "description",
      "type": "string"
    }
  ]
}

That should make managing these individual config files easier, as you will have IntelliSense on them.

@estruyf
Copy link
Owner

estruyf commented Sep 30, 2022

The sample was updated to support custom scripts and data files, folders, and types.

The following settings are now supported to be split in multiple files:

Setting name JSON Schema Folder path
frontMatter.content.pageFolders https://beta.frontmatter.codes/config/content.pagefolders.schema.json ./frontmatter/config/content/pagefolders/
frontMatter.content.placeholders https://beta.frontmatter.codes/config/content.placeholders.schema.json ./frontmatter/config/content/placeholders/
frontMatter.content.snippets https://beta.frontmatter.codes/config/content.snippets.schema.json ./frontmatter/config/content/snippets/
frontMatter.custom.scripts https://beta.frontmatter.codes/config/custom.scripts.schema.json ./frontmatter/config/custom/scripts/
frontMatter.data.files https://beta.frontmatter.codes/config/data.files.schema.json ./frontmatter/config/data/files/
frontMatter.data.folders https://beta.frontmatter.codes/config/data.folders.schema.json ./frontmatter/config/data/folders/
frontMatter.data.types https://beta.frontmatter.codes/config/data.types.schema.json ./frontmatter/config/data/types/
frontMatter.taxonomy.contentTypes https://beta.frontmatter.codes/config/taxonomy.contenttypes.schema.json ./frontmatter/config/taxonomy/contenttypes/

The sample (https://github.com/FrontMatter/project-samples/tree/main/docusaurus) got updated to test and show how this functionality works.

@estruyf
Copy link
Owner

estruyf commented Sep 30, 2022

Documentation has been updated as well with more information: https://beta.frontmatter.codes/docs/settings#splitting-your-settings-in-multiple-files

michaeltlombardi added a commit to michaeltlombardi/flagrant-garden-site that referenced this issue Oct 2, 2022
Taking advantage of the preview for estruyf/vscode-front-matter#412,
this commit decomposes the previously monolithic `frontmatter.json` file
by moving the definitions for `pageFolders`, `snippets`, and
`contentTypes` into their own files in the `.frontmatter/config` folder.

This should make future diffs and maintenance much simpler. The primary
configuration file now reads primarily as _configuration_ for the site's
behavior with Front Matter, rather than defining the dozens of data
types and helpers for various users.

This model will also allow us to more readily define the data types and
folders, which would've previously added hundreds of lines to the
primary configuration.
michaeltlombardi added a commit to michaeltlombardi/flagrant-garden-site that referenced this issue Oct 2, 2022
Prior to this change, the data files for the project needed to be
updated by hand with the available schemas.

With this change, again taking advantage of the preview feature for
splitting the config made available in estruyf/vscode-front-matter#412,
we now have the various Flagrant Factions data types and their data
entry files incorporated to Front Matter.

This will allow us to modify those values in a nice UI instead of
manually in YAML.
@michaeltlombardi
Copy link

Just decomposed more than 700 lines of configuration - this is much more maintainable for me and if/when #407 becomes more feasible, it will make for vastly improved maintainability for "packaged" configurations.

@michaeltlombardi
Copy link

Just ran across something possibly worth addressing - as I started to do some work today on my theme/helpers to make it easier to configure sites for my users, I noticed that my data types definitions in particular were getting a little out of hand and thought it would be good to break them into subfolders inside of `.frontmatter/config/data/types.

However, this doesn't seem to work - anything in a subfolder does get processed, but the file doesn't show up in the data view.

Screenshot of configuration files and dashboard

I suspect it's to do with how the file names/settings are being parsed:

// Get the path without the filename
const configFolder = parseWinPath(dirname(configFilePath));
let relSettingName = configFolder.split('/').join('.');
if (relSettingName.startsWith('.')) {
relSettingName = relSettingName.substring(1);
}
relSettingName = relSettingName.toLowerCase();

else if (relSettingName === SETTING_DATA_FILES.toLowerCase()) {
Settings.updateGlobalConfigArraySetting(SETTING_DATA_FILES, "id", configJson);
}
// Data folders
else if (relSettingName === SETTING_DATA_FOLDERS.toLowerCase()) {
Settings.updateGlobalConfigArraySetting(SETTING_DATA_FOLDERS, "id", configJson);
}
// Data types
else if (relSettingName === SETTING_DATA_TYPES.toLowerCase()) {
Settings.updateGlobalConfigArraySetting(SETTING_DATA_TYPES, "id", configJson);
}

It looks like the relSettingName for .frontmatter/config/data/types/bar/baz.json becomes data.types.bar while .frontmatter/config/data/types/foo.json becomes data.types and therefore works.

Would it be possible to see instead if we could use something like

if (relSettingName.includes(SETTING_DATA_TYPES.toLowerCase())) {
  ...
}

So that if the path includes the correct key, we can assume it belongs to that key?

Alternatively, if the file includes a $schema key, maybe it could use that key to determine where to insert the settings?

For example, this definition (in .frontmatter/config/data/types/bar/baz.json)

{
  "$schema": "https://beta.frontmatter.codes/config/data.types.schema.json",
  "id": "hugo.params.baz",
  "schema": {
    "title": "Baz Site Parameters for hugo-toroidal",
    "type": "object",
    "properties": {
      "First": {
        "title": "First Property",
        "description": "First Baz",
        "type": "string",
        "default": ""
      }
    }
  }
}

could be reasonably inferred to map to data.types given the value of $schema, I think - though I don't know the performance cost here vs using the folder structure.

@estruyf
Copy link
Owner

estruyf commented Oct 8, 2022

@michaeltlombardi indeed, sub-folders are, as you mentioned, not working, as I currently linked it to that particular folder structure.

The logic can be changed to support sub-folders. The performance should still be fine.

Not sure if the schema would be the best, as this is not required, and not sure if everyone would use it.

@estruyf
Copy link
Owner

estruyf commented Oct 8, 2022

  • Make sure the frontmatter.json file wins over the folder - as this makes local changes easier, instead of searching in the folders
  • Allow sub-folder usage

@michaeltlombardi
Copy link

Not sure if the schema would be the best, as this is not required, and not sure if everyone would use it.

I realize I didn't explicitly say this, but I was thinking of that check as a fallback. With subfolder support though I don't think it's necessary.

@michaeltlombardi
Copy link

I also just realized, I'll make a PR to the docs for clarifying that you can check the composed settings with the Diagnostic logging command.

estruyf added a commit that referenced this issue Oct 31, 2022
@estruyf
Copy link
Owner

estruyf commented Oct 31, 2022

@michaeltlombardi sub-folder should now be supported, feel free to give it a try

michaeltlombardi added a commit to michaeltlombardi/flagrant-garden-site that referenced this issue Nov 4, 2022
Prior to this change, the decomposed Front Matter configuration files
were restricted to being directly in the folder for the config type they
represented, requiring paths like `data/types/hugo.params.toroidal.json`
for namespacing configuration items in an organized way.

With sub-folder support added, this commit reorganizes the configuration
files, enabling a more normalized (and collapsible) folder-subfolder
structure.

See estruyf/vscode-front-matter#412 for details on the implementation.
@michaeltlombardi
Copy link

Works a treat, extremely happy with being able to use subfolders to organize the data bits! I'm now looking over my snippets and having a think about those, because that list is also growing long, but I know that the naming for the snippets maps directly to their file name.

I think any changes here might have to be addressed in a different way, if at all. 🤔 I don't exactly want "folders" of snippets in the UI I think, but there's also no way to contend with having a bunch of snippets in the existing UI (as addressed in #440).

@estruyf
Copy link
Owner

estruyf commented Nov 4, 2022

Would it help to have a title field in the snippet? That way, if not set or used, it will use the file as key, otherwise it will override it will the title from within the snippet.

@michaeltlombardi
Copy link

That would help enormously and mean I don't have to have spaces in file names 😅

It would also bring snippets more in line with the UX surface of other configurable items.

@estruyf
Copy link
Owner

estruyf commented Nov 7, 2022

I will also need to change how to edit the snippets. For now, when they are external, they get a new action to open the file instead of editing these.

image

estruyf added a commit that referenced this issue Nov 7, 2022
@estruyf
Copy link
Owner

estruyf commented Nov 7, 2022

🚀 you can now add titles to your snippets, which makes most sense when adding snippets by individual files, not when you add these within your settings directly.

@michaeltlombardi
Copy link

Rad!!

michaeltlombardi added a commit to michaeltlombardi/flagrant-garden-site that referenced this issue Nov 7, 2022
This change takes advantage of further improvements in
estruyf/vscode-front-matter#412 to reorganize the snippets into folders
and subfolders for easier maintenance, adding the new `title` key for
each.
estruyf added a commit that referenced this issue Nov 14, 2022
@estruyf estruyf closed this as completed Dec 8, 2022
@estruyf estruyf mentioned this issue Dec 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Project: v8.2.0
Projects
None yet
Development

No branches or pull requests

3 participants