About texture and image data formats

[javagl]

Sorry, this is likely not a real "issue". And it's hard to phrase this as a clear question. But after having created basic glTF viewers in Java and JavaScript, there is still this uncomfortable feeling of not having understood the intended image- and texture handling properly.

There are various pieces of information related to textures and image data formats in the discussions of previous issues, and in some issues that are still open ( #620 , #640 ...). I have to admit that I did not yet wrap my head around all the relevant points of these discussions, and likely am not as deeply familiar with some details and best practices in OpenGL/WebGL/JavaScript as I should be.

---

A texture basically contains several GL constants, and a reference to an image:

"textures": {
    "exampleTexture": {
        "target": GL_TEXTURE_2D,
        "internalFormat": GL_RGBA,
        "format": GL_RGBA,
        "type": GL_UNSIGNED_BYTE,
        "source": "exampleImage",
        "sampler": "exampleSampler"
    }
}
"images": {
    "exampleImage": {
        "uri": "example.png"
    }
},

The constants are exactly the parameters that are supposed to be passed to texImage2D.

(Fortunately, in WebGL 1, the format and the internalFormat must be the same. This simplifies things, considering the can of worms that is opened with format conversions ).

As the last parameter, this function receives the image data that was read from the source image of the texture. From my understanding, in JavaScript, the image data will usually (but not necessarily) be an ImageData object. According to the documentation, the data that may be obtained from this ImageData object is...

.. a Uint8ClampedArray representing a one-dimensional array containing the data in the RGBA order, with integer values between 0 and 255 (included).

So I assume the following: When the ImageData version of this function is used, then the format/type have to be GL_RGBA/GL_UNSIGNED_BYTE. Is this correct?

Conversely, and what mainly confuses me here: You may not even know in which format/type you receive the image data. Particularly, the data may simply "appear" in a format that is different from the one that is given in the texture.

As an example, I'm thinking about Java here. You can write

BufferedImage image = ImageIO.read(new File("example.png"));

and then you don't know (but have to check) whether the image data type is

TYPE_3BYTE_BGR
TYPE_INT_ARGB
TYPE_4BYTE_ABGR
...

or some other type. This depends on several factors, e.g. whether the image contains transparency etc. Similarly, there might be an image loading library in JavaScript or C++ that "by default" delivers the image data as GL_RGBA/GL_UNSIGNED_INT or GL_BRG/GL_UNSIGNED_BYTE.

How is this supposed to be handled by a viewer? Wouldn't it be legitimate to simply ignore the format/type, and just make sure that the image data at hand is somehow uploaded to GL?

---

I know, the type information will be crucial for "custom" image types and more generic image sources (i.e. images that are not read from PNG/JPG files). But for the common case, I wonder whether one should take the detour of converting image data that was read e.g. as INT_ARGB into the format that is given in the texture, when the outcome is always the same: The image data can be used by OpenGL, regardless of how it got there...

[lexaknyazev]

As the last parameter, this function receives the image data that was read from the source image of the texture. From my understanding, in JavaScript, the image data will usually (but not necessarily) be an ImageData object.

Last parameter has TexImageSource type; it's defined like this:

typedef (ImageBitmap or
         ImageData or
         HTMLImageElement or
         HTMLCanvasElement or
         HTMLVideoElement) TexImageSource;

Usually, runtime passes HTMLImageElement directly. User-agent (web browser) knows about actual image format (RGB or RGBA). Then,

The source image data is conceptually first converted to the data type and format specified by the format and type arguments, and then transferred to the WebGL implementation.

[javagl]

OK, maybe I'll have to read more in different docs here, but until now, I fetched the ImageData with a function roughly like http://stackoverflow.com/a/10755011 , and then passed this as the last argument to texImage2D - but think I only tried this with textures that already had GL_RGBA/GL_UNSIGNED_BYTE, and not with different formats.

The quoted passage from the WebGL spec seems to say that this should work even when the type is not RGBA, and that the texImage2D call does the possibly necessary conversions automagically on the fly. I'll probably just try it out when I'm back at my dev PC.

However, for non-JavaScript, non-Browser-based environments, I assume that the format/type information from the texture can be ignored, and the image data can just be uploaded to GL in whatever way it is appropriate for the type that it actually has, right?

[lexaknyazev]

If you have a variable of HTMLImageElement, you can use it directly:

var img = document.getElementById('myimg');
gl.texImage2D(target, 0, internalformat, format, type, img);

WebGL will convert data to the specified format/type.

However, for non-JavaScript, non-Browser-based environments, I assume that the format/type information from the texture can be ignored, and the image data can just be uploaded to GL in whatever way it is appropriate for the type that it actually has, right?

For PNG/GIF/JPG images type depends on the used decoder output. format could depend on the number of channels in actual image (RGB/RGBA).

For proper support of traditional GPU texture formats, texture (and probably image) objects should be refined (also see #620 and #640).

[javagl]

Thanks for these inputs. Regarding the particular case of JavaScript, I'll have to read more about how this plays together (or can be combined with) asynchronous operations. My gut feeling until is that using the HTMLImageElement will block the rendering process (at the texImage2D call) until the image is downloaded - but that's a different story. I'll try to figure out some further details about these automagic conversions in texImage2D later.

But if I understood this correctly, it is valid to ignore the format/type of the texture, when the "real" format of the image data is known. (That's also what I'm currently doing in JglTF: Convert the image to INT_ARGB, and use fixed parameters for glTexImage2D, ignoring the format/type)

[lexaknyazev]

My gut feeling until is that using the HTMLImageElement will block the rendering process

Much simpler: texture2D will return (0, 0, 0, 1). You should call texImage2D, when image is loaded (e.g., in img.onload event handler).

But if I understood this correctly, it is valid to ignore the format/type of the texture, when the "real" format of the image data is known.

Both format and type could affect rendering:

• if you use RGB format on RGBA image, you'll have all-opaque data in shader;
• you could downsample color depth from 24-bit to 16-bit (by providing other type).

[lexaknyazev]

Closing as there's no action items here. Advanced texture transmission will be covered by CTTF/KTX2.