Sixel Graphics Implementation by CannibalVox · Pull Request #352 · charmbracelet/x

CannibalVox · 2025-02-01T18:13:17Z

This PR adds support for sixels to x/ansi. Sixels are a protocol for writing images to the terminal by writing a large blob of ANSI-escaped data. They function by encoding columns of 6 pixels into a single character (in much the same way base64 encodes data 6 bits at a time). Sixel images are paletted, with a palette established at the beginning of the image blob and pixels identifying palette entries by index while writing the pixel data.

Sixels are written one 6-pixel-tall band at a time, one color at a time. For each band, a single color's pixels are written, then a carriage return is written to bring the "cursor" back to the beginning of a band where a new color is selected and pixels written. This continues until the entire band has been drawn, at which time a line break is written to begin the next band.

Sixel writing and reading take the form of the sixel.Encoder and sixel.Decoder types, in order to match the existing kitty graphics implementation as closely as we reasonably can. The only issue is that because there is no way to know the size of a sixel blob in advance (unlike the kitty graphics protocol, we can't determine the size of the payload from the width and height), we can only determine the boundaries of the payload by detecting the ST sequence (or BEL or whatever we're using to close the payload). As a result, the sixel decoder accepts a byte slice that contains only the payload and relies on the caller to determine the boundaries.

I included an options type in the parameters for WriteSixelGraphics, but there are not yet any options. I'd like to include a dithering implementation in the future, but I wanted to get the actual sixel implementation checked in first.

aymanbagabas

Impressive work here!

Initial review here with few comments to abstract and generalize the implementation. I wonder if we need a sixel.Options here, instead, we can place encoder/decoder customizations and options directly in the structure definitions.

aymanbagabas · 2025-02-01T21:10:46Z

+	"strconv"
+	"strings"
+
+	"github.com/bits-and-blooms/bitset"


Is there anyway we can get rid of this dependency?

The only way I see it working is writing our own bitset implementation- is that something you want to pursue?

aymanbagabas · 2025-02-01T21:23:47Z

+// sixelColor is a flat struct that contains a single color: all channels are 0-100
+// instead of anything sensible
+type sixelColor struct {
+	Red   uint32
+	Green uint32
+	Blue  uint32
+	Alpha uint32
+}


Suggested change

// sixelColor is a flat struct that contains a single color: all channels are 0-100

// instead of anything sensible

type sixelColor struct {

Red uint32

Green uint32

Blue uint32

Alpha uint32

}

// sixelColor is a flat struct that contains a single color: all channels are 0-100

// instead of anything sensible

type RGBAColor color.RGBA

Are we good with the fact that RGBA colors sourced from outside this package will not work properly?

RGBA colors that are not sixel.RGBAColor can be converted using sixelConvertColor

Thinking more about this, I think it makes more sense if we have type Color struct { Pc, Pu uint8; Px, Py Pz int } and functions the same as the other types, func (Color) Encode(io.Writer) error, func (Color) String() string, and func ParseColor([]byte) (Color, error)

aymanbagabas · 2025-02-01T21:26:05Z

+//   - If a single color sits on a cut line, all pixels of that color are assigned to one of the subcubes
+//     rather than try to split them up between the subcubes.  This allows us to use a slice of unique colors
+//     and a map of pixel counts rather than try to represent each pixel individually.
+type sixelPalette struct {


Should we expose this and be able to pass a custom Palette in *Options?

aymanbagabas · 2025-02-01T21:29:24Z

+func (e *Encoder) encodePaletteColor(w io.Writer, paletteIndex int, c sixelColor) {
+	// Initializing palette entries
+	// #<a>;<b>;<c>;<d>;<e>
+	// a = palette index
+	// b = color type, 2 is RGB
+	// c = R
+	// d = G
+	// e = B
+
+	w.Write([]byte{sixelUseColor})                //nolint:errcheck
+	io.WriteString(w, strconv.Itoa(paletteIndex)) //nolint:errcheck
+	io.WriteString(w, ";2;")
+	io.WriteString(w, strconv.Itoa(int(c.Red)))   //nolint:errcheck
+	w.Write([]byte{';'})                          //nolint:errcheck
+	io.WriteString(w, strconv.Itoa(int(c.Green))) //nolint:errcheck
+	w.Write([]byte{';'})
+	io.WriteString(w, strconv.Itoa(int(c.Blue))) //nolint:errcheck
+}


Let's define sixel.BasicColor, sixel.RGBAColor, sixel.HLSColor types that implement color.Color and fmt.Stringer. We can also have a sixel.ParseColor([]byte) (color.Color, error) to decode a color from bytes.

What is a basiccolor in this case, is it just getting a palette index with no color information?

aymanbagabas · 2025-02-01T21:33:38Z

+type Decoder struct {
+}


Could we make the palette customizable?

Suggested change

type Decoder struct {

}

type Decoder struct {

Palette *Palette

}

Does this just cover default colors, or should we also ignore color definitions when a palette is provided?

We use default colors if Palette is nil

Right, but do the passed in palette colors get overwritten by defined colors, or do they only replace the default colors?

aymanbagabas · 2025-02-01T22:08:02Z

Tests are failing because the ansi package targets go1.18 which doesn't support the cmp and slices packages

Co-authored-by: Ayman Bagabas <ayman.bagabas@gmail.com>

CannibalVox · 2025-02-02T16:43:23Z

I think I should probably mention: the encoder previously used a single stringbuilder in order to minimize allocations. These Raster/Repeat/Color/etc. methods will each add an allocation every time they're called, which will impact performance.

aymanbagabas · 2025-02-02T20:17:58Z

I think I should probably mention: the encoder previously used a single stringbuilder in order to minimize allocations. These Raster/Repeat/Color/etc. methods will each add an allocation every time they're called, which will impact performance.

Fair point. Let's change these methods to be writers WriteRaster(io.Writer, ...) (int, error) etc

aymanbagabas · 2025-02-13T15:02:47Z

 }

+func ParseRepeat(data io.Reader) (count int, r byte, err error) {
+	_, err = fmt.Fscanf(data, "%d%b", &count, &r)


I would take a []byte over a reader for simplicity and so that it works with raw bytes

aymanbagabas · 2025-02-18T20:31:38Z

+	var buf bytes.Buffer
+
+	buf.WriteString("\x1bP")
+	buf.WriteString(strconv.Itoa(p1))


Suggested change

buf.WriteString(strconv.Itoa(p1))

if p1 >= 0 {

buf.WriteString(strconv.Itoa(p1))

}

aymanbagabas · 2025-02-18T20:32:32Z

+	buf.WriteString("\x1bP")
+	buf.WriteString(strconv.Itoa(p1))
+	buf.WriteByte(';')
+	buf.WriteString(strconv.Itoa(p2))


Suggested change

buf.WriteString(strconv.Itoa(p2))

if p2 >= 0 {

buf.WriteString(strconv.Itoa(p2))

}

aymanbagabas · 2025-02-18T20:32:52Z

+	buf.WriteByte(';')
+	buf.WriteString(strconv.Itoa(p2))
+	buf.WriteByte(';')
+	buf.WriteString(strconv.Itoa(p3))


Suggested change

buf.WriteString(strconv.Itoa(p3))

if p3 >= 0 {

buf.WriteString(strconv.Itoa(p3))

}

aymanbagabas · 2025-02-18T20:38:17Z

+func Raster(pan, pad, ph, pv int) string {
+	return fmt.Sprintf("%s%d;%d;%d;%d", string(RasterAttribute), pan, pad, ph, pv)
+}


Perhaps we can have a type Raster struct { Pan, Pad, Ph, Pv int} with func (Raster) Encode(io.Writer) error, func (Raster) String() string that uses r.Encode, and func ParseRaster([]byte) (Raster, error)

aymanbagabas · 2025-02-18T20:58:55Z

+func Repeat(count int, repeatRune rune) string {
+	var sb strings.Builder
+	sb.WriteByte(RepeatIntroducer)
+	sb.WriteString(strconv.Itoa(count))
+	sb.WriteRune(repeatRune)
+	return sb.String()
+}


Same, a type Repeat struct { Count int; Char byte } with func (Repeat) Encode(io.Writer) error, func (Encode) String() string, and func ParseRepeat([]byte) (Repeat, error)

aymanbagabas

With these types reflecting the sixel graphics protocol, we could drop the sixel.Options type and embed these types in sixel.Decoder and sixel.Encoder

aymanbagabas · 2025-02-18T21:07:35Z

+// sixelColor is a flat struct that contains a single color: all channels are 0-100
+// instead of anything sensible
+type sixelColor struct {
+	Red   uint32
+	Green uint32
+	Blue  uint32
+	Alpha uint32
+}


Thinking more about this, I think it makes more sense if we have type Color struct { Pc, Pu uint8; Px, Py Pz int } and functions the same as the other types, func (Color) Encode(io.Writer) error, func (Color) String() string, and func ParseColor([]byte) (Color, error)

j4james · 2025-02-18T21:49:30Z

+// p3 = Horizontal grid size parameter. Everyone ignores this and uses a fixed grid
+// size, as far as I can tell.


Just FYI, this p3 parameter was only used by Sixel printers. It never applied to Sixel terminals because their screen pixels were obviously in a fixed position, so there was no way to change the horizontal spacing.

raphamorim · 2025-02-19T15:55:03Z

fyi this implementation perf comparisson with go-sixel

goos: darwin
goarch: arm64
pkg: github.com/charmbracelet/tapioca
cpu: Apple M4 Pro
BenchmarkGoSixel-14               6     179179312 ns/op
BenchmarkXSixel-14               31      36864684 ns/op
PASS
ok      github.com/charmbracelet/tapioca    2.644s

comparing encoding only:

package abs

import (
    "os"
    "bytes"
    "fmt"
    "testing"
    "image"
    "image/png"

    "github.com/mattn/go-sixel"
    "github.com/charmbracelet/x/ansi"
    
    sixxel "github.com/charmbracelet/x/ansi/sixel"
)

func BenchmarkGoSixel(b *testing.B) {
    for b.Loop() {
        raw, err := loadImage("./sixel.png")
        if err != nil {
            os.Exit(1)
        }

        var b = bytes.NewBuffer(nil)
        enc := sixel.NewEncoder(b)
        if err := enc.Encode(raw); err != nil {
            fmt.Fprintln(os.Stderr, err)
            os.Exit(1)
        }

        // fmt.Println(b)
    }
}

func BenchmarkXSixel(b *testing.B) {
    for b.Loop() {
        raw, err := loadImage("./sixel.png")
        if err != nil {
            os.Exit(1)
        }

        var b = bytes.NewBuffer(nil)
        if err := ansi.WriteSixelGraphics(b, raw, &sixxel.Options{}); err != nil {
            fmt.Fprintln(os.Stderr, err)
            os.Exit(1)
        }
    }
}

func loadImage(path string) (image.Image, error) {
    f, err := os.Open(path)
    if err != nil {
        return nil, err
    }
    return png.Decode(f)
}

raphamorim · 2025-02-19T16:05:15Z

Memory, although it was tested with 154kb image.

goos: darwin
goarch: arm64
pkg: github.com/charmbracelet/tapioca
cpu: Apple M4 Pro
BenchmarkGoSixel-14               6     180208833 ns/op     7961162 B/op      607769 allocs/op
BenchmarkXSixel-14               31      37303151 ns/op    35627236 B/op      546321 allocs/op
PASS
ok      github.com/charmbracelet/tapioca    2.677s

raphamorim · 2025-02-19T16:09:03Z

With 10.9mb png image (encoding only)

goos: darwin
goarch: arm64
pkg: github.com/charmbracelet/tapioca
cpu: Apple M4 Pro
BenchmarkGoSixel-14    	       1	6574080208 ns/op	333589664 B/op	30287393 allocs/op
BenchmarkXSixel-14     	       1	1017670666 ns/op	975905416 B/op	19713507 allocs/op
PASS
ok  	github.com/charmbracelet/tapioca	8.046s

raphamorim · 2025-02-21T22:52:52Z

Hey @CannibalVox! We’ve pulled your changes into #380 and built on them a bit 🙏

Thank you very much for the contribution @CannibalVox and making Sixel possible in x !

CannibalVox added 2 commits January 30, 2025 08:26

feat: Add sixels to ansi package

ed43f81

scanSize tests

d44fac5

CannibalVox requested a review from aymanbagabas as a code owner February 1, 2025 18:13

CannibalVox mentioned this pull request Feb 1, 2025

feat: Add SixelImage charmbracelet/lipgloss#436

Closed

aymanbagabas reviewed Feb 1, 2025

View reviewed changes

CannibalVox and others added 5 commits February 1, 2025 15:46

Update ansi/sixel/encoder.go

16b7248

Co-authored-by: Ayman Bagabas <ayman.bagabas@gmail.com>

Update ansi/graphics.go

f672bee

Co-authored-by: Ayman Bagabas <ayman.bagabas@gmail.com>

pr comments

9734d1f

raster/parseraster

41a4558

repeat/parserepeat

8705df7

aymanbagabas reviewed Feb 13, 2025

View reviewed changes

aymanbagabas reviewed Feb 18, 2025

View reviewed changes

j4james reviewed Feb 18, 2025

View reviewed changes

raphamorim mentioned this pull request Feb 19, 2025

feat: add sixel to ansi package #380

Merged

aymanbagabas mentioned this pull request Feb 20, 2025

Implement Sixel encoder/decoder support #381

Open

raphamorim closed this Feb 21, 2025

		// p3 = Horizontal grid size parameter. Everyone ignores this and uses a fixed grid
		// size, as far as I can tell.

Conversation

CannibalVox commented Feb 1, 2025

Uh oh!

aymanbagabas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

aymanbagabas commented Feb 1, 2025

Uh oh!

CannibalVox commented Feb 2, 2025

Uh oh!

aymanbagabas commented Feb 2, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aymanbagabas left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

raphamorim commented Feb 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

raphamorim commented Feb 19, 2025

Uh oh!

raphamorim commented Feb 19, 2025

Uh oh!

raphamorim commented Feb 21, 2025 • edited by meowgorithm Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

raphamorim commented Feb 19, 2025 •

edited

Loading

raphamorim commented Feb 21, 2025 •

edited by meowgorithm

Loading