"Extending Twine" suggestion: a preprocessing function for engine code and engine CSS

[webbedspace]

This is another feature that has been on my mind for awhile: I think there should be additional (optional) inserts added to the story format spec called {{STORY_CODE}} and {{STORY_CSS}}, which would be filled by the results of new story format preprocessor functions that would be passed information about what kind of build is currently happening (debug vs normal, the locale, etc). This would allow the following amazing features:

• Not including Debug Mode code + CSS in final releases of people's games (Harlowe's debug mode has gotten a tad meatier over the years).
• Being able to include author-specific localisation strings for Debug Mode/error messages at compile time (instead of the "copy-paste your preferred localisation code into every one of your stories" method currently recommended on SugarCube's page).
• If the full story data of the current story was passed to these functions, then more aggressive optimisation could be implemented: Macros and CSS features that are verifiably never used could have their definitions dropped from the engine JS, for instance.

I do not believe any of these are really possible with the present state of the specifications.

Obviously this would involve changing both the Story Format Spec (although that's been out-of-date since 2.4 released) and Extending Twine.

Other possibilities

An alternative specification idea would be to have "source" be either a string or a function (taking the aformementioned build data), but that would be a little messier for implementors to use (insofar as the function would have to return the story's HTML as well as its JS/CSS - plus, it would kind of throw into question the purpose of {{STORY_DATA}} etc. in the first place, if this function can be used to ignore it entirely).

Presubmission checklist

• I am interested in working on code that would implement this feature request. (This is not required to submit a suggestion.)
• I have done a search and believe that an issue does not already exist for this idea in the GitHub repository.
• I have read and agree to abide by this project's Code of Conduct.

[klembot]

Interesting idea. I think locale and trimming debug features out are reasonable use cases.

I'd be curious as to what other story format authors think would be a good approach here. My main interest is in keeping any mechanism as simple as possible, and optional if possible... it's hard enough as it is to compile a story format.

[tareksander]

I'm writing a story format that could use some preprocessing or a build step, for performance and code size: I want to include Typescript, Svelte and and Sass support. I could just include the compilers in the story format, but that would bloat it unnecessarily. My proposed solution to the interop with e.g. Twine compilers that can't run JS is to just add an optional build function: If it's there and the compiler supports it, it gets used. Otherwise, the old source property gets used, and the story format has to make sure it also works like that, even if it's bigger or slower.

[klembot]

@tareksander appreciate you sharing your use case here. Do you have anything I could look at to understand your goals better, like a project page?

[tareksander]

I'm still in early development and working on the basics of story loading, rendering and saving.

I want to include Typescript errors in CodeMirror with typescript-eslint, and transpile to JS using Babel standalone. To make interactive UIs easier than they are in SugarCube, I want to include the Svelte framework, which also has its own compiler. And to offer reusable styles, I want to use sass. I'd rewire the imports of Typescript and SCSS to use passages, to make both modular. Without a build step, I'd have to include the Babel transpiler, the Svelte compiler and the Sass compiler in the story format source, which wouldn't really be needed and adds unnecessary size overhead. Also I'd need to transpile and compile everything on every load (load time overhead), or cache it in local storage (size overhead).

My goal is essentially to make a new story format with similar features to SugarCube, but with integrated Typescript support and better interactive UIs. The Typescript would stay in the story JS or passages with the "typescript" tag, same with SCSS and a "scss" tag. Svelte would use a "svelte" tag and compile the components in to web components you can include in the passages. I may also need to use a module bundler or implement my own lightweight module instantiation and caching, Webpack doesn't run in the browser, Rollup seems to be an option.

For CodeMirror errors and completion, I'd need something like #942 to be able to analyze the imports.

I made my own Twee compiler in Rust, but it'd be nice if my story format could support Twine itself instead of just my custom compiler.

I'd be willing to see if I can implement the features I need in Twine btw, I have some JS experience and I wouldn't need to touch UI stuff anyways.

[tareksander]

Other goals are translatability (wrapping a text block in e.g. double curly braces and giving it an ID, and have passages where you can map the ID to strings of other languages, with the original as fallback) and offline capability with a service worker. It'd be nice if the build function could generate additional files for a published zip file like I mentioned in #114 just now, e.g. to include a web manifest and a service worker JS file (which needs to be a separate file for technical reasons, sadly).

[klembot]

I think let's talk again when your project has solidified a little bit more. My instinct right now is that people who are writing Svelte, SASS, and TypeScript are going to want a full-blown editor experience akin to VS Code or other IDEs, with features like LSP integration, and that kind of thing is out of scope for Twine. But that's just a gut feeling and not based on evidence. I agree including all those compilers into a runtime is not feasible.

A possible road forward is you could try forking Twine for this, and if you find success there, we could look at incorporating features from a fork here.

[tareksander]

After thinking about it a bit more, I think you're right: I should probably build my own engine and make a VSCode extension with a Twine-like UI for editing stories. That way I can easily have my build step, and I don't need to make language extensions for CodeMirror.

[klembot]

@tareksander if you haven’t already, take a look at https://marketplace.visualstudio.com/items?itemName=cyrusfirheir.twee3-language-tools. I think it could be a useful thing to combine with what you’re working on.

[tareksander]

I actually used the Twee language definitions from that extension in my own Twine IDE extension, I basically added support for building and playing the story in VSCode. For my own engine I'll probably not use Twee, but just Markdown, as that's already widely supported in editors.