Unable to correctly drive 3-color display (InkyWHAT)

[alanta]

I'm trying to use my Pimorony InkyWHAT (black/yellow) with CircuitPython. I've created a driver for it based on the code from Pimoroni.
The code is here: alanta/CircuitPython_InkyWhat

The driver mostly works but it seems that the display needs it's data in different format. Here's the expected output:

This is the actual output:

I've tried all combinations of color_bits_inverted and black_bits_inverted which gives different output but all incorrect.

The original Pimoroni driver sets the black plane to be 1 where the color != white, the color plane is 1 where the color is the accent color. Is there some way to get the same result?