Abusing the Sixel support in Windows Terminal for fun and no-profit

[mrange]

Hi.

Since Sixel support was added in Windows Terminal by @j4james and @DHowett I have had fun tinkering with it.

That culminated in a Windows Terminal demo I submitted for a demo competition this weekend (got 2nd). I thought some here would appreciate this odd use case for Sixel Support in Windows Terminal :)

https://youtu.be/004ng70swbY

(Sixeling starts at about 1:15 if you don't want to see the classic terminal effects).

[j4james]

I cannot express how mind-blowingly awesome this is! Reminds me of the days when I used to download demos from a BBS over a painfully slow modem connection. Congratulations on your placing in the competition!

I just want to point out that much of the credit for our Sixel support is due to @lhecker, who implemented the DirectX renderer, as well as the conpty passthrough functionality, without which we wouldn't have been able to get Sixel in Windows Terminal.

[lhecker]

Without exaggerating, this is the nicest thing I've ever seen anyone build on top of something I've worked on in my career. Thank you so much!

I'm surprised that it works as well as it does. I did not really spend any time on intentional performance optimizations for Sixels or its integration into our text renderer. What makes me incredibly happy as well is seeing the last animation at the end: https://www.youtube.com/watch?v=004ng70swbY&t=196s
I spent a lot of time fine tuning those builtin glyphs and even now I'm not fully happy with them yet. But seeing it used in such a beautiful way makes me just so so happy.

---

By the way, and this is only loosely related (to those builtin glyphs), but I recently attempted to implement a tiled, deferred text renderer using a compute shader. My goal is to fully solve the glyph overlap issue.

It works! I don't know yet when I can finish it and submit a PR as I'm currently working on a yet-to-be-announced project, but hopefully it should be within this year.

[j4james]

how do I present page 2 when I am done with? I enable DECPCCM?

Yeah, when you enable DECPCCM, it'll make the active page become the visible page. Your sample code above looks perfect to me.

[mrange]

Thanks alot!

[mrange]

Just FYI: Seems to work. The Hello World is painted above the sixel graphics and at least on this computer now flickering.

Another weird wish is to make the background transparent :)

[j4james]

The Hello World is painted above the sixel graphics

In case you aren't aware, this is dependent on the order of output. If you output the text first, the graphics will be on top. In this particular case you wouldn't then see anything, but if there were transparent areas in the image, the underlying text would show through those holes.

Another weird wish is to make the background transparent :)

Me too! There's actually a DEC "overstrike" mode we could potentially support that would enable this, but I'm not sure how feasible it is. My one idea was to treat overstrike text as an image operation, and just write straight into our image layer. That would probably be easiest, but the quality might not be great, because it'd be limited to our emulated VT340 resolution.

Another idea was to have a limited overstrike support which only allowed for two layers (the normal text layer, plus an additional layer of text on top of that). That would require the buffer being capable of managing that additional content, tracking the z order when mixed with image content, and extra complexity in the renderer to draw it all.

So as much I'd like this functionality myself, I'm not sure there's enough demand for it to justify the overhead. But @lhecker is the expert when it comes to buffering and rendering, so he'd probably be the best person to answer that question.

Btw, I should also mention that Mintty actually already supports an overstrike mode using something like the two layer approach I discussed above. Unfortunately their implementation doesn't currently work with images.

[mrange]

Thanks for all the info.

[DHowett]

I cannot overstate how cool this is. Seriously, thank you for sharing it!

[mrange]

Offtopic: Do you have some Windows Terminal ASCII Art? Not too big. I want to test something :)