Support YUY2-encoded images #4877

oxkitsune · 2024-01-20T13:01:57Z

What

This PR Introduces support for YUV422/YUY2 encoded images based on #3541.

Image showcasing RGB, NV12 and YUV422 encoded images

The raw (encoded) image data is stored in an R8Uint texture. The contents of the texture get decoded in the rectangle_fs.wgsl via the decoder written in decodings.wgsl.

Other YUV image formats could be added easily, following the NV12 implementation with slight modifications to the decoder.

Checklist

I have read and agree to Contributor Guide and the Code of Conduct
I've included a screenshot or gif (if applicable)
I have tested the web demo (if applicable):
- Using newly built examples: app.rerun.io
- Using examples from latest main build: app.rerun.io
- Using full set of examples from nightly build: app.rerun.io
The PR title and labels are set such as to maximize their usefulness for the next release's CHANGELOG

oxkitsune · 2024-01-20T13:03:22Z

I want to improve the rust api for logging images with a different encoding than the traditional rgb and resolve #4820 in the process, but I'm not sure what the best approach would be for this.

One thought is to implement some helper methods on TensorData e.g. TensorData::from_yuv422. WDYT?

crates/re_types/src/datatypes/tensor_data_ext.rs

Wumpf

Nice work! Can go in as-is once we figured out the CI kinks, if you want I can take care of it.

The only wrinkle here is that we nowhere specify that this is the YUY2 variant (YuYv instead of uYvY) for the format. I don't know how common the other one is, but https://en.wikipedia.org/wiki/YCbCr#4:2:2 mentions both.

Also, just like with NV12 before we don't have these documented anywhere. I tried to find a nice spot to put doctext in the fbs files, but codegen isn't picking that up yet, so we'll have to do this as a follow-up.
It's also a bit sad how many places needs to be touched to get this going, something to re-conciliate for us in the future.
That said, as an extension to status quo this looks perfect 👌

crates/re_renderer/shader/decodings.wgsl

oxkitsune · 2024-01-24T09:27:24Z

Nice work! Can go in as-is once we figured out the CI kinks, if you want I can take care of it.

I'll sort it asap, I want to get my local environment set up so I can easily fix CI stuff in the future, seems like my python formatter is producing some different output.

The only wrinkle here is that we nowhere specify that this is the YUY2 variant (YuYv instead of uYvY) for the format. I don't know how common the other one is, but en.wikipedia.org/wiki/YCbCr#4:2:2 mentions both.

Yea, @YukumoHunter also mentioned this. The gist linked in #3541 specifically used YUV422, so I based it on that. I think renaming it to YUY2 over YUV422 might be better? WDYT?

Wumpf · 2024-01-24T09:31:17Z

Hmm yeah let's rename it to YUY2 everywhere to proof against adding the other variant in the future. Thank you!
Might be good to document that in the fbs file where the format is "first" mentioned already despite that not showing up in generated code just yet

Co-Authored-By: Harold <37922636+yukumohunter@users.noreply.github.com>

oxkitsune · 2024-01-24T23:08:56Z

Alright, I've renamed it to YUY2 and @YukumoHunter cleaned up the colour space conversions. I think this is ready now 👍

Wumpf · 2024-01-25T09:50:43Z

crates/re_renderer/shader/decodings.wgsl

+    } else {
+        // we're on an odd pixel, so we need to sample the second y value
+        // we add 2 to the column to get the second y value
+        y = f32(textureLoad(texture, vec2u(uv_col + 2u, uv_row), 0).r);


Didn't notice either at first, but this offset is wrong actually. We already muliplied the uv_col with 2, so it points at the Y value right now. I found out when adding yuy2 to the (former) nv12 python test script which also highlighted that there was still some stuff missing in ImageEncoded

To illustrate:
With that offset:

Without that offset:

Wumpf

Great work and extra thanks for further cleaning up the conversion code both in wgsl and Rust!

When trying to test this I found that extension for ImageFormat etc. was still missing and once I added that it looked a bit off, highlighting that there was a small mistake in the offset calculation for Y.
I think it's all good now and ready to go, but can you please confirm first with your own dataset that this checks out?

oxkitsune · 2024-01-25T11:13:47Z

Just tested it out, and it seems to be working nicely! Thanks!

oxkitsune · 2024-01-25T11:18:17Z

On closer look, seems like there is a small issue though. Might be an issue with our dataset though, I'll quickly test this with live data when I get to the lab in a few hours.

Wumpf · 2024-01-25T11:41:32Z

Hmm looks like the row is offset every second pixel or so, strange!
btw. worth pointing out that the test image I used was a quasi-manual conversion since opencv didn't offer it, see here https://github.com/rerun-io/rerun/pull/4877/files#diff-3d8a8e06114b0f435bfd964a239e55fa6d7f8ef6308276e632a36d5f62af03e5R28

Wumpf · 2024-01-25T14:26:26Z

not entirely sure what's up with the python wheel builds, but it's obviously not this pr's fault. Things work for me locally so what could possibly go wrong. Time to merge!

… and avoid crash on chrome (#3931) (#5074) ### What When we added support for YUY2 format in (#4877) we we re-triggered the issue originally originally reported in #3931 and now #5073 - This is a more extreme version of: #3948 which was the workaround the first time we hit this. ### Checklist * [x] I have read and agree to [Contributor Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and the [Code of Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md) * [x] I've included a screenshot or gif (if applicable) * [x] I have tested the web demo (if applicable): * Using newly built examples: [app.rerun.io](https://app.rerun.io/pr/5074/index.html) * Using examples from latest `main` build: [app.rerun.io](https://app.rerun.io/pr/5074/index.html?manifest_url=https://app.rerun.io/version/main/examples_manifest.json) * Using full set of examples from `nightly` build: [app.rerun.io](https://app.rerun.io/pr/5074/index.html?manifest_url=https://app.rerun.io/version/nightly/examples_manifest.json) * [x] The PR title and labels are set such as to maximize their usefulness for the next release's CHANGELOG * [x] If applicable, add a new check to the [release checklist](tests/python/release_checklist)! - [PR Build Summary](https://build.rerun.io/pr/5074) - [Docs preview](https://rerun.io/preview/e35e3a58e1f2093140d4d63479098124f6556463/docs)  - [Examples preview](https://rerun.io/preview/e35e3a58e1f2093140d4d63479098124f6556463/examples)  - [Recent benchmark results](https://build.rerun.io/graphs/crates.html) - [Wasm size tracking](https://build.rerun.io/graphs/sizes.html)

oxkitsune commented Jan 20, 2024

View reviewed changes

crates/re_types/src/datatypes/tensor_data_ext.rs Outdated Show resolved Hide resolved

oxkitsune force-pushed the gdjeong/yuv422 branch 2 times, most recently from 166a90c to abbb72a Compare January 20, 2024 14:22

abey79 added 🔺 re_renderer affects re_renderer itself include in changelog labels Jan 20, 2024

oxkitsune marked this pull request as ready for review January 23, 2024 01:12

Wumpf self-requested a review January 23, 2024 19:39

Wumpf approved these changes Jan 24, 2024

View reviewed changes

crates/re_renderer/shader/decodings.wgsl Show resolved Hide resolved

oxkitsune changed the title ~~Support YUV422-encoded images~~ Support YUY2-encoded images Jan 24, 2024

oxkitsune and others added 9 commits January 25, 2024 00:01

Add support for YUV422 encoded images

de1ec06

Co-Authored-By: Harold <37922636+yukumohunter@users.noreply.github.com>

make ci happy

d7d5fbd

appease clippy

812a6c8

py-format

dbb4faa

run codegen

a7dcfac

Rename YUV422 -> YUY2

86e2f38

update shader const

8d4ffd3

mirror set_color_standard between rust and shader code

38c3563

run codegen

3e8f48a

oxkitsune force-pushed the gdjeong/yuv422 branch from e9b75dc to 3e8f48a Compare January 24, 2024 23:05

Wumpf self-requested a review January 25, 2024 08:54

fix link formatting to make clippy happy

313a581

Wumpf reviewed Jan 25, 2024

View reviewed changes

Wumpf added 4 commits January 25, 2024 10:54

fix bug in yuy2 decoding

c28d5c2

fix nv12 sample swapping bgr with rgb

fe97105

fix tensordata size validation for yuy2, support YUY2 in ImageFormat

19c08ba

add yuy2 to chroma downsample test

a38d05d

Wumpf approved these changes Jan 25, 2024

View reviewed changes

Wumpf added 2 commits January 25, 2024 11:00

fix python linter unhappiness

b5b47ca

fix python linter unhappiness

a74762a

Use direct BGRA -> YUY2 conversion in example

3f787b4

Wumpf merged commit 50671b2 into rerun-io:main Jan 25, 2024
32 of 35 checks passed

This was referenced Feb 6, 2024

Intermittent crashes / hangs on chrome on ubuntu #5073

Open

Make rectangle_fs.wgsl compile on chrome despite angle/mesa bug (#3931) #5074

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support YUY2-encoded images #4877

Support YUY2-encoded images #4877

oxkitsune commented Jan 20, 2024 •

edited by github-actions bot

oxkitsune commented Jan 20, 2024

Wumpf left a comment

oxkitsune commented Jan 24, 2024

Wumpf commented Jan 24, 2024

oxkitsune commented Jan 24, 2024

Wumpf Jan 25, 2024

Wumpf left a comment

oxkitsune commented Jan 25, 2024

oxkitsune commented Jan 25, 2024

Wumpf commented Jan 25, 2024

Wumpf commented Jan 25, 2024

Support YUY2-encoded images #4877

Support YUY2-encoded images #4877

Conversation

oxkitsune commented Jan 20, 2024 • edited by github-actions bot

What

Checklist

oxkitsune commented Jan 20, 2024

Wumpf left a comment

Choose a reason for hiding this comment

oxkitsune commented Jan 24, 2024

Wumpf commented Jan 24, 2024

oxkitsune commented Jan 24, 2024

Wumpf Jan 25, 2024

Choose a reason for hiding this comment

Wumpf left a comment

Choose a reason for hiding this comment

oxkitsune commented Jan 25, 2024

oxkitsune commented Jan 25, 2024

Wumpf commented Jan 25, 2024

Wumpf commented Jan 25, 2024

oxkitsune commented Jan 20, 2024 •

edited by github-actions bot