Suggest breaking changes for v4

[jonpyt]

I'm planning to begin working on version 4. It won't be a major rewrite, just some small breaking changes. If anyone has suggestions for breaking changes they'd like to see, leave them here.

[panoply]

PCE is evolving rapidly, and I'm currently using it as a drop-in replacement for Prism. While my variation significantly extends the existing implementation, I use PCE to generate both static and editor output code blocks.

Hover Descriptions

I think it would be great to introduce hover descriptions. The addition of completions is excellent, and we could take it a step further by supporting hover descriptions in isolation, though this requires some consideration.

Recently, I found the need to effectively describe some code within a code block. Personally, I don't like Shiki much, but I do appreciate the two-slash implementation. However, it comes with a lot of overhead and can be difficult to manage. It's definitely not something for PCE, but a variation could work quite well, in my opinion.

A very rough example of how this might look:

{
  onHover({ token: string, language: string }) {
    
     // token will be the grammar token name
     // language will be the current language name
    
     return {
        description: string; // provide an HTML string to render in the tooltip
        signature: string;  // provide an HTML string to render as the signature
     };
  }
}

While this example doesn't account for inside grammar structures, the general idea is to trigger a callback when the pointer hovers over a specific token within a rendered code block. The return type signals to PCE to render a description and/or signature in a tooltip popover within the editor.

The idea here is to allow the user to specify the hovers to apply, without doing too much heavy lifting. The current CSS and styling give us great customisation options, the same can apply with something like this, meaning the user can control how the tooltip/popover looks.

Hydration

We briefly touched on this in GitHub discussion #21. I currently have an approach that works well but has plenty of room for improvement. As mentioned, I'm using PCE both as an editor provider and for static (readonly) code blocks. I'll describe my current setup to provide context if you decide to explore this further.

In my case, when using PCE for highlighting in a Node environment, I pass it code output from a tool like markdown-it. I perform syntax highlighting using the available tokenizeText and highlightTokens exports, which return the Prism-highlighted markup string.

It's important to note that in some cases, I don't want to include PCE in my browser bundle, as I only need it for highlighting (via Prism). However, I still like to have some features, like line numbers. To achieve this, I augment the returned highlightTokens string as follows:

const tokenize = tokenizeText('...', language);
const rawCode = highlightTokens(tokenize);

// Insert line numbers
const lineNumbers = rawCode
      .split('\n')
      .map((token, i) => `<div class="line-no" aria-hidden="true" data-line="${i + 1}">${token}</div>`)
      .join('');

While this adds line numbers, if I wanted to use an extension like copyButton, I would need to include PCE in my browser bundle and initialize an editor instance for each static code block, which isn't ideal. Instead, it would be preferable to have a partial export that works with static (readonly) code blocks.

FWIW: I currently use a small inline utility to add features like a copy button while also statically rendering <div class="overlays">.

---

The Future of PCE

I just want to reiterate my appreciation for what you've accomplished with this project. You've built something impressive, and it's clear how much thought has gone into it. It's been great thus far being able to use PCE in my own projects. That said, while the underlying Prism parsing algorithm is powerful, it does come with inherent limitations. I would advocate and champion for exploring alternative grammar-level approaches to bypass some of the constraints imposed by Prism's parsing strategy.

As PCE continues to evolve and new features are introduced, these limitations may become more pronounced, potentially creating performance bottlenecks. I'd be interested to know if you've considered other approaches to tackle these issues or if you're committed to maintaining the current structure, and if you have or are considering a different route on that front, I'd love to help.

[jonpyt]

I think it would be great to introduce hover descriptions.

I have thought about this, but the name reason I haven't implemented it is that mouse events are captured by the textarea, and the syntax highlighted code has pointer-events: none. There is a solution to this, and it involves Document.elementsFromPoint(). I would have to attach a mousemove event listener on the textarea that removes pointer-events: none from the content before calling elementsFromPoint() to track which tokens are hovered at all times. I suspect the performance of this won't be amazing, but it's probably fine if the DOM isn't massive. I haven't thought about whether or not this will work with touch devices.

The extension could take a function that renders the description by either returning a string or a DOM node. This function could receive the language, an array containing the token name, along with its aliases, the tokens textContent, and the DOM node for the token. I see that you're proposing the function returns a description, and a signature. Could you explain what signature would be? I believe returning a DOM node is better and more flexible.

I think hovers should only be triggered on tokens that only have text, and that a class should be added to hovered tokens.

There's also the issue where multi-line tokens are split into multiple DOM nodes, but this is probably not a big deal.

Hydration

I have already implemented this on a new branch. I can publish the branch now if you want to have a look. I'm working on creating a new documentation website for PCE using starlight, and it's been awesome so far using the "hydration" method we discussed. It should work really well for what you're doing too.

Code blocks

It's important to note that in some cases, I don't want to include PCE in my browser bundle, as I only need it for highlighting (via Prism). However, I still like to have some features, like line numbers.

I have been thinking about adding static code blocks as well. I've figured out how indentation can be preserved on wrapped lines. I dislike word wrap, but with indentation preserved, it's much better. This can never be added to editors unfortunately, since it would cause misalignment with the textarea. Next, I can add guide indents to the code blocks by adding a repeating-linear-gradient to the lines and adding background-size as an inline style for each line to control how many guides are shown. This would support word wrap as well.

Not sure how overlays could be added to code blocks. The copy buttons aren't visible unless the code block is hovered, so it would be wasteful to have them in the HTML, and JS is needed anyway to make them clickable unless we use event handler attributes. Maybe a function could be exported that adds specified overlays to code blocks in the client?

[jonpyt]

So, I looked more into the hover descriptions. Changing the pointer-events property requires style calculations. Doing this on every mousemove event is not a great idea. The performance decreases roughly linearly with the number of tokens in the editor since the computed value of pointer-events changes twice for every token.

However, the hover descriptions could be added to the code blocks I've proposed since here mouse events are dispatched on the tokens directly. Let me know your thoughts.

[panoply]

Hey! Sorry for the late response—my daytime job has been hectic this time of year. I'd love to take a look at what you've got so far. I have PCE integrated into Papyrus (locally) and have made some extensive changes to fit my preferences. One of the great things about PCE is its API. Having used it extensively, I’m always pleased to see how well it handles edge cases. Anyway, I digress.

Could you explain what a signature would be?

Initially, I considered exposing a subset of rendering options, probably influenced by LSP usage. In hindsight, a simpler and more practical approach, as you mentioned, is to use either a string or a DOM node.

Code Blocks

Note: I refer to "code blocks" as "statics."

Static code blocks are essential in my opinion. The current workaround I’m using is far from elegant, so I’m eager to explore a more polished solution. One feature I frequently use is inline highlighting. Here’s a screenshot from a side project where `{js} function(){}` results in <code> inline highlighting (notice <head> and <script spx-eval="false"> applying inline highlights):

Note: While unfinished, I have PCE handling static code block highlighting in this project and also customizing grammars. As I mentioned before, the API is fantastic—features like embeddedIn are especially useful.

Hover Descriptions

Personally, I’m happy to limit this feature to static code blocks only and exclude it from editor mode. In fact, I feel it could complicate the quick, drop-in nature of PCE (with PrismJS grammars). However, working on a solution for editor mode would be interesting as a challenge. I wonder if handling this in editor mode could be done in dormant states—just thinking out loud.

For my use cases, hover descriptions in static code blocks would make it much easier to communicate the functionality of code compared to using comments.

Overlays on Code Blocks

Perhaps we should only expose a subset of overlays? A function to handle this sounds ideal. However, there could be a consideration for using IIFE invocation based on attribute annotations for static code blocks. Either way, it’s not a must-have for me.

Hydration

I use 11ty together with markdown-it for static sites and haven't worked with Astro, so I can’t provide feedback until I get a better understanding of its architecture and how I can adapt it. Currently, I handle things by rendering a PCE structure statically, adding a data-* attribute. At runtime on the client side, I grab all static (readonly) PCE occurrences and pass them to an editor to invoke a PCE instance.

---

Still interested if you have given thought on exploring alternative grammar-level approaches...?

[jonpyt]

The v4 branch is public!

Code blocks

I've added the code blocks you asked for with features like indentation preservation on wrapped lines, line numbers using CSS counters, and indentation guides. They look great IMO. Here's the code that renders a code block as an HTML string. The code blocks require an additional stylesheet from prism-code-editor/code-block.css to work correctly.

Overlays

I decided to go for a function that would run a callback function for each code block under a specified root. Here's that function. You can add a copy button by calling this function inside the callback.

Hover descriptions

Personally, I’m happy to limit this feature to static code blocks only and exclude it from editor mode.

Yup, adding this to editors isn't easy to do in a performant manner. I have added it to code blocks though, and it works quite well. Here's the code for it. I would be careful putting critical information in hover descriptions however, since they won't be accessible to keyboard-only users unless we add a tabindex to every token (which is a terrible idea).

Inline highlighting

One feature I frequently use is inline highlighting.

Don't think this library needs to provide helpers for this. Just create a <code> element, and sets its HTML to the syntax highlighted snippet. This is just a few lines of code.

Hydration

The method I went with is incredibly simple. Render an editor as a static HTML string on the server with renderEditor(), and on the client, you can mount all editors under a specified root with mountEditorsUnder(). This should be usable with almost any static site generator that runs JavaScript. Something I came up with was for renderEditor() to serialize any additional properties to its configuration object as JSON. This JSON is then parsed by mountEditorsUnder(), so you can easily customize how extensions are added by passing extra data from the server. You might've seen in the code that the code blocks support something similar.

The new documentation website using this method for rendering editors is close to finised, so I'll probably push it to the v4 branch in the next few days if you want to run it locally. I wouldn't mind some feedback on it 🤔.

Still interested if you have given thought on exploring alternative grammar-level approaches...?

I haven't felt held back by Prism much honestly, it's a great syntax highlighter. The performance bottleneck is the DOM, not the syntax highlighting, and I'm happy with its features. I've considered using native lookbehind assertions for better performance in some situations, but I don't think we're quite there in terms of browser support yet (thanks Safari). I've also thought about adding a custom tokenizer to JSDoc comments that would remove the asterisk at the start of each line and insert them back into the token stream later. This would allow for highlighting code blocks in JSDoc comments for example, but I figured it was rather unnecessary.

One last thing. I'm considering renaming most CSS variables by prefixing them with pce-, but I'm hesitant to do so because I think those who have created custom themes would find it annoying to rename most CSS variables: What do you think?