Enhancement: 😀 Support for YAML non-printing characters

[estruyf]

Related to the following Discord thread: https://discord.com/channels/992409023607476224/1085255612939640832

Very often, the YAML front matter includes text that will display on a page, like the following:

title: Hello World 🌎

However, the proper YAML would be:
title: Hello World \U0001F4CD

FM currently adds the non-printable character (like an emoji) to the text, which errors out on build since it is not properly formatted YAML

I can understand that FM would not want to modify an existing file, but there is an opportunity to catch non-printable characters when creating/modifying metadata in the Front Matter rail

Author: @BillRaymond

[estruyf]

Adding the encodeEmoji property to the string field. When enabling this property on the field, it encodes all emojis.

Example: Hello World 🌎 becomes Hello World \u1F30E.

Verified against: https://unicodeplus.com/U+1F30E

[davidsneighbour]

Isn't

title: "Hello World 🌏"

possible/valid too?

[estruyf]

Yes @davidsneighbour, but for @BillRaymond it isn't it seems. That is why an extra property is introduced. If your SSG/framework wouldn't support it, you can let FM encode it for you.

[estruyf]

By default, it will keep using Hello World 🌏, only if you need to encode the emoji, you will have to tell FM to do this.

[BillRaymond]

Isn't

title: "Hello World 🌏"

possible/valid too?

While you can add emojis and other non-printable characters to YAML, they are technically not allowed. See here. Some YAML processors handle this for you, but others do not. In my specific situation, I use Jekyll (also the default for GitHub Pages), which errors out. That is why I requested that FM provide a conversion option.

[BillRaymond]

Thank you!

[BillRaymond]

Since you are still working on this enhancement, I did some testing on my local machine using Front Matter. I noticed you only sometimes add quotations around strings. However, you need the quotes if you add escape codes to display non-printable characters.

For example:

title: \U0001F30E The title of my post

Outputs:

\U0001F30E The title of my post

But, if you put the title in quotes, you get the non-printable character. For example:

title: "\U0001F30E The title of my post"

Outputs:

🌎 The title of my post

I hope this will help you during your development process.

[estruyf]

Thanks @BillRaymond, it is the YAML parser which decides to add quotes or not, will take a look why and when.

[estruyf]

I have done some tests, and in my environment, when double quotes are added, it will use a double backslash, which is how it should be to correctly parse the string.

Screen.Recording.2023-03-21.at.09.51.32.mov

[BillRaymond]

In this example, I add the Unicode escape character into the title. FM does not automatically put the title in quotes.

If the YAML is not in quotes, the Unicode is processed as regular text and does not display the non-printable character.

However, if I manually put the title in quotes, the proper non-printable character is processed and displays on the website.