Runtime Specification

[tcannonfodder]

This issue is to kick off a project discussed in the Inkle discord with @ephread: writing a specification for Ink runtimes.

Reason

Right now, building an ink runtime requires reverse-engineering the C# implementation, which causes a few issues:

• Increased engine developer burden
• (Likely) differences between runtimes, making ports harder
• Code that will feel like the C# implementation, rather than writing a language/platform-specific implementation that matches the existing ecosystem

How we can help

If runtime developers collaborate & write a specification for:

• The high-level architecture of an Ink runtime
• A comprehensive spec for the compiled JSON bytecode
• How to parse & prepare the JSON bytecode from Ink exports
• How state saving & restoration should work
• How external function calls work in Ink
• Accompanying tests, including a defined "call and response" test suite that plays through a complete, compiled Ink story over various paths to verify the behavior ( https://github.com/inkle/the-intercept feels like a great candidate for this)

We can dramatically improve Ink's tooling, and pave the way for better adoption across the board; both for current & future tools.

[smwhr]

I totally agree with all your reasons and I think that any kind of specification is a sign of a mature language and will insure the perennity of all our tools for the years to come.

One big advantage of the "transparent C#" ports so far has been that it was very easy to upgrade them by looking at the commits from Inkle's team and just copying them over, making maintenance and sync with the original implementation an easy task once the the bulk of the code has been translated. I agree that this first step is indeed a heavy burden for anyone trying to port to any new language.
With a spec, when changes are made to the core C# code, this would mean understanding and interpreting the changes, amending the spec, and then applying it to the local codebases. I know, this is how it's done in the real world :D

( https://github.com/inkle/the-intercept feels like a great candidate for this)

From the admission of the inkle's team, I think that "The Intercept" is quite old in its idioms and way of organising the story, and does not take advantage of any real advanced ink features (iirc, this is a straight export from inklewriter). Any recent recent examples (like the ones that are in the book : eg : pontoon) should do the job though.

Are there specification document (for say, other game engine or runtime) whose form you think will fit for this particular project ? (for some reason the spec of the z-machine came to my mind)

[tcannonfodder]

I agree that this first step is indeed a heavy burden for anyone trying to port to any new language.

It also adds friction for new maintainers on a runtime, because they have to:

1. Reverse-engineer the C# codebase in order to gain a foundational understanding
2. Understand the runtime implementation (because no translation is 100% perfect)
3. Reverse-engineer any changes they want to work on

With a spec, step 1 would be easier, since there would be written explanations, and tests to boot. Plus all the ancillary benefits of having step 2 be language-tailored, and Step 3 being much more discrete & easier to break up across maintainers.

Also, with a specification, there can be a (slight) delay between what the C# code has, and what other runtimes implement. Ideally, the delay would be 0, but if we have versioned specifications a runtime can decide what level of support works for their needs.

Any recent recent examples (like the ones that are in the book : eg : pontoon) should do the job though.

Works for me! For context: I don't have a lot of free time, but I believe in Ink as a tool and both want to see it succeed and build my own runtimes for my own games. I got very close with https://github.com/tcannonfodder/fable , but ran into a combination of low bandwidth outside of work, and quirks I couldn't figure out from the C# implementation. Hence my desire to help out with a spec!

Are there specification document (for say, other game engine or runtime) whose form you think will fit for this particular project?

I don't have any specific, but I have a ton of experience both reading and writing API documentation (and API design documentation) as part of my career. I designed, implemented, and wrote the documentation for: https://developer.nokotime.com

I'll dig into the z-machine documentation as I have time! I think the first step is to write a specification of all the data types that the runtime has. That will help get the foot in the door, and give us something to start working with.

[SirMetathyst]

I'm writing a port for golang atm and don't know how this runtime works so I'm really depending on there being enough (good) tests... A specification would indeed be nice. I actually want to build a website for ink stories powered by go but also (as a side note) concerned about pulling a massive ink file from db, processing it and then disposing it each time (unless I can cache it in user session on server side but that will take up alot of memory... need to look into chunking up the file somehow)

[vincentkerdraon]

@SirMetathyst I am very interested to follow your project.
In go, I had the same problem and I fixed it in a different way. But I would prefer to direct go implementation.

See https://github.com/vincentkerdraon/inkcaller.
What I do is I create a JS VM each time to run my code. This is performant enough for my need but I would migrate if something better was to be available.
Note I don't have exactly the same need, because I inject the full ink state at each call.

(My project is public but I don't think it is stable enough to release it yet. I am still working on a game using this as a dependency and I will probably change the API in the future.)

[SirMetathyst]

@vincentkerdraon Wow, cool project and It looks like it works which is more than I can say for mine (which is not public yet).

My implementation is very early (even the few constructs I've written are only partially complete except Path/PathComponent which pass the test I found in the test suite) and I'm trying to replicate as close to the c# logic as possible so that I increase my chances of it just working later...

Unfortunately there are very few unit tests in the c# runtime targeted at low level components (path, container, etc) and mostly concentrate at Story level so alot of go code will be written before I can actually test to see what works properly...

I've never written a port but I'll finish writing some stuff which I'm still converting then post my project on github...

I've been meaning to get round to it. Ultimately when/if I manage to get this thing working I'd like to write up as many tests as I can manage before rewriting it again in a hopefully better design or at least clean it up (I've been finding the c# version to be a bit of mess imo). Thanks for your interest.

[SirMetathyst]

@tcannonfodder @smwhr I think the ink runtime itself could serve as a specification in and of itself if we clean up the codebase and wrote up additional unit tests for each construct. I don't think we can really write up a proper spec without doing first? Or it should make writing a spec easier should we do so first.

Some things we might want to do:

• removing the operator overloading which allows you to do (x) instead of (x != null) which although nice is pretty useless syntactic sugar imo for more than ~50 lines of code).
• Remove/Change/Move function(s)/properties which casts to Container in Object despite Container being a subclass of Object. What's going on there?
• Possibly move Json and SimpleJson out of the runtime itself.. (not investigated into this)

[SirMetathyst]

@vincentkerdraon I've created the project, you can find it here: https://github.com/SirMetathyst/go-ink

[tcannonfodder]

@tcannonfodder @smwhr I think the ink runtime itself could serve as a specification in and of itself if we clean up the codebase and wrote up additional unit tests for each construct. I don't think we can really write up a proper spec without doing first? Or it should make writing a spec easier should we do so first.

I agree with these changes, but getting them implemented into the main branch might not be feasible in a reasonable timeline, because they’d introduce breaking changes to the Ink API (which could seriously mess up buildchains).

Even if it’s suboptimal, we can still write a spec around these implementation quirks; clearly labeling them in a specification, and shrugging at how the C# implementation is written (which, is kinda the beauty of specs. As long as it passes spec, folks can write however works best for them)

[tcannonfodder]

Also; as an update: Still want to work on this, I’ve just been absolutely swamped so I haven’t had the capacity to yet

[tcannonfodder]

@smwhr @SirMetathyst I've started a fork for the documentation, let me know if you'd like to be added as collaborators! https://github.com/tcannonfodder/ink

[tcannonfodder]

PR to break up the work a bit: #749

[tcannonfodder]

To prevent ourselves from writing a specification against a moving target, we should write the specification based on the code in a published release. In this case, 1.0: https://github.com/inkle/ink/releases/tag/v1.0.0

I know it's likely we'll miss certain bug fixes, but we should be able to establish the foundation by working against the 1.0.0 release. And bugfixes should be point releases on the specification; so this would help nudge the project towards using releases rather than directly pulling from the main branch

[SirMetathyst]

@vincentkerdraon in the end it might be better for my project to use the InkJs runtime and just transmit the json (I was intending to use go-ink to support no-js as well but state retention and restoring may really complicate the backend), however I'll continue on the go-ink and see how far I get with it.

[biter777]

@SirMetathyst your go-ink project in working order? Is it usable?

[SirMetathyst]

@biter777 no. It has some bugs in it preventing it from working correctly. The entire reason for porting was to have a website written in go to host ink stories but I’ve since realised it’s better to write a new language targeting the web with built in markdown or markdown like support. Feel free to debug and complete it if you wish.

[biter777]

@SirMetathyst
Is your new language an analog or replacement for ink?
Does it support writing quests?

[SirMetathyst]

@biter777 I'm in the progress of a rewrite at the moment and experimenting with its syntax design but it's basic concept is that it works like a graph where you begin at a main node and you transfer to other nodes rather than call them, rather than adding to a call stack like ink tunnels since it makes save/load more unstable. Although I do want functions I'm focused on

• if statements
• nodes and transferring to other nodes
• global variables
• markdown like formatting: links, lists, task lists, images, tables etc.

Its powered by a bytecode register based vm where It's just a long array I load up and process saving the state after buffered content output. I can then direct load the next instruct on next continue from browser.

But I don't want to add/do too much too quickly. I also have a basic cli to build, run and preview in a web server. I'll also rewrite that. What id like to do is get a solid base or foundation working with some tooling etc and more to the language where necessary.

Are you interested in collaborating over discord about it? My username is sirmetathyst.