Transparent background for 3d model scene

[kulnaman]

Hi,
Thanks for making such an awesome library. I am using this library in android and trying to render a 3d model. The model is getting rendered correctly but the camera preview is getting obfuscated by a black screen. I have used the camera's view and projection matrix, how can I use the camera image data as the scene background or is it possible to make the scene background transparent?

[javagl]

I'm afraid that I don't understand what you mean by "obfuscated with a black screen".

The library does not contain an android renderer by default. There is #34 , but it's not really integrated yet. So I'm curious how you are currently rendering. Did you implement an own GlContext using android, or are you using an entirely different rendering process?

(The description sounds like it could have something to do with glClear, but until now, that's only a wild guess...)

[kulnaman]

I'm currently using the https://github.com/mikikg/AndroidJgltfViewer for rendering the gltf model. My main aim is to render a 3d model in an AR app, for this we are using maxst framework, where a custom background render is used to render camera image data,
This is the background renderer class.

public class Yuv420_888Renderer extends BackgroundRenderer {
	private static final String VERTEX_SHADER_SRC =
			"attribute vec4 a_position;\n" +
					"uniform mat4 u_mvpMatrix;\n" +
					"attribute vec2 a_vertexTexCoord;\n" +
					"varying vec2 v_texCoord;\n" +
					"void main()\n" +
					"{\n" +
					"   gl_Position = u_mvpMatrix * a_position;\n" +
					"   v_texCoord = a_vertexTexCoord;          \n" +
					"}\n";

//
private static final String FRAGMENT_SHADER_SRC =
		"precision mediump float;\n" +
		"uniform sampler2D u_texture_1;\n" +
		"uniform sampler2D u_texture_2;\n" +
		"uniform sampler2D u_texture_3;\n"							+
		"varying vec2 v_texCoord;\n" +
		"void main()\n" +
		"{\n" +
		"    float y = texture2D(u_texture_1, v_texCoord).r;\n" +
		"    float v = texture2D(u_texture_2, v_texCoord).a;\n" +
		"    float u = texture2D(u_texture_3, v_texCoord).a;\n" +
		"    y = 1.1643 * (y - 0.0625);\n" +
		"    u = u - 0.5;\n" +
		"    v = v - 0.5;\n" +
		"    float r = y + 1.5958 * v;\n" +
		"    float g = y - 0.39173 * u - 0.81290 * v;\n" +
		"    float b = y + 2.017 * u;\n" +
		"    gl_FragColor = vec4(r, g, b, 1.0);\n" +
		"}\n";


	private static final float[] VERTEX_BUF = {
			-0.5f, 0.5f, 0.0f,
			-0.5f, -0.5f, 0.0f,
			0.5f, -0.5f, 0.0f,
			0.5f, 0.5f, 0.0f
	};

	private static final short[] INDEX_BUF = {
			0, 1, 2, 2, 3, 0
	};

	private static final float[] TEXTURE_COORD_BUF = {
			0.0f, 1.0f,
			0.0f, 0.0f,
			1.0f, 0.0f,
			1.0f, 1.0f,
	};

	private ByteBuffer yBuffer;
	private ByteBuffer uBuffer;
	private ByteBuffer vBuffer;

	Yuv420_888Renderer() {
		super();
		ByteBuffer bb = ByteBuffer.allocateDirect(VERTEX_BUF.length * Float.SIZE / 8);
		bb.order(ByteOrder.nativeOrder());
		vertexBuffer = bb.asFloatBuffer();
		vertexBuffer.put(VERTEX_BUF);
		vertexBuffer.position(0);

		bb = ByteBuffer.allocateDirect(INDEX_BUF.length * Integer.SIZE / 8);
		bb.order(ByteOrder.nativeOrder());
		indexBuffer = bb.asShortBuffer();
		indexBuffer.put(INDEX_BUF);
		indexBuffer.position(0);

		bb = ByteBuffer.allocateDirect(TEXTURE_COORD_BUF.length * Float.SIZE / 8);
		bb.order(ByteOrder.nativeOrder());
		textureCoordBuff = bb.asFloatBuffer();
		textureCoordBuff.put(TEXTURE_COORD_BUF);
		textureCoordBuff.position(0);

		shaderProgramId = ShaderUtil.createProgram(VERTEX_SHADER_SRC, FRAGMENT_SHADER_SRC);

		positionHandle = GLES20.glGetAttribLocation(shaderProgramId, "a_position");
		textureCoordHandle = GLES20.glGetAttribLocation(shaderProgramId, "a_vertexTexCoord");
		mvpMatrixHandle = GLES20.glGetUniformLocation(shaderProgramId, "u_mvpMatrix");

		textureNames = new int[3];
		textureHandles = new int[3];

		GLES20.glGenTextures(3, textureNames, 0);
		for (int i = 0; i < 3; i++) {
			GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, textureNames[i]);
			GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MIN_FILTER, GLES20.GL_LINEAR);
			GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_MAG_FILTER, GLES20.GL_LINEAR);
			GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_WRAP_S, GLES20.GL_CLAMP_TO_EDGE);
			GLES20.glTexParameteri(GLES20.GL_TEXTURE_2D, GLES20.GL_TEXTURE_WRAP_T, GLES20.GL_CLAMP_TO_EDGE);
			String textureHandleId = "u_texture_" + (i + 1);
			textureHandles[i] = GLES20.glGetUniformLocation(shaderProgramId, textureHandleId);
		}
	}

	private int yDataLength = 0;
	private int uDataLength = 0;
	private int vDataLength = 0;

	public void draw(TrackedImage image) {

		if (image.getData() == null || image.getWidth() == 0) {
			return;
		}

		int yDataLength = image.getWidth() * image.getHeight();
		int uDataLength = image.getWidth() * image.getHeight() / 2 - 1;
		int vDataLength = image.getWidth() * image.getHeight() / 2 - 1;

		if (this.yDataLength != yDataLength || this.uDataLength != uDataLength || this.vDataLength != vDataLength) {
			this.yDataLength = yDataLength;
			this.uDataLength = uDataLength;
			this.vDataLength = vDataLength;

			yBuffer = ByteBuffer.allocateDirect(yDataLength);
			uBuffer = ByteBuffer.allocateDirect(uDataLength);
			vBuffer = ByteBuffer.allocateDirect(vDataLength);
			yBuffer.order(ByteOrder.nativeOrder());
			uBuffer.order(ByteOrder.nativeOrder());
			vBuffer.order(ByteOrder.nativeOrder());
		}

		yBuffer.put(image.getData(), 0, yDataLength);
		yBuffer.position(0);

		uBuffer.put(image.getData(), yDataLength, uDataLength);
		uBuffer.position(0);

		vBuffer.put(image.getData(), yDataLength + uDataLength, vDataLength);
		vBuffer.position(0);

		GLES20.glUseProgram(shaderProgramId);

		GLES20.glVertexAttribPointer(positionHandle, 3, GLES20.GL_FLOAT, false,
				0, vertexBuffer);
		GLES20.glEnableVertexAttribArray(positionHandle);

		GLES20.glVertexAttribPointer(textureCoordHandle, 2, GLES20.GL_FLOAT, false,
				0, textureCoordBuff);
		GLES20.glEnableVertexAttribArray(textureCoordHandle);

		Matrix.multiplyMM(localMvpMatrix, 0, projectionMatrix, 0, transform, 0);
		GLES20.glUniformMatrix4fv(mvpMatrixHandle, 1, false, localMvpMatrix, 0);

		GLES20.glActiveTexture(GLES20.GL_TEXTURE0);
		GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, textureNames[0]);
		GLES20.glTexImage2D(GLES20.GL_TEXTURE_2D, 0, GLES20.GL_LUMINANCE, image.getWidth(), image.getHeight(), 0,
				GLES20.GL_LUMINANCE, GLES20.GL_UNSIGNED_BYTE, yBuffer);
		GLES20.glUniform1i(textureHandles[0], 0);

		GLES20.glActiveTexture(GLES20.GL_TEXTURE1);
		GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, textureNames[1]);
		GLES20.glTexImage2D(GLES20.GL_TEXTURE_2D, 0, GLES20.GL_LUMINANCE_ALPHA, image.getWidth() / 2, image.getHeight() / 2, 0,
				GLES20.GL_LUMINANCE_ALPHA, GLES20.GL_UNSIGNED_BYTE, uBuffer);
		GLES20.glUniform1i(textureHandles[1], 1);

		GLES20.glActiveTexture(GLES20.GL_TEXTURE2);
		GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, textureNames[2]);
		GLES20.glTexImage2D(GLES20.GL_TEXTURE_2D, 0, GLES20.GL_LUMINANCE_ALPHA, image.getWidth() / 2, image.getHeight() / 2, 0,
				GLES20.GL_LUMINANCE_ALPHA, GLES20.GL_UNSIGNED_BYTE, vBuffer);
		GLES20.glUniform1i(textureHandles[2], 2);

		GLES20.glDrawElements(GLES20.GL_TRIANGLES, INDEX_BUF.length,
				GLES20.GL_UNSIGNED_SHORT, indexBuffer);

		GLES20.glDisableVertexAttribArray(positionHandle);
		GLES20.glDisableVertexAttribArray(textureCoordHandle);
		GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, 0);
		GLES20.glUseProgram(0);
	}
}

So my issue is how can i integrate this background render with the gltf Renderer ?

[javagl]

Admittedly, I'd need a refresher for some parts of the GL code, particularly the shader parts (I haven't worked much with GL lately...).

On top of that, I'd have to make a more detailed review of the changes that are done in the AndroidJgltfViewer project. Some of the changes there apparently are not done ~"how an Android support was intended to be implemented from my point of view", but that's, entirely my fault: It was caused by (maybe a lack of documentation, and) the fact that Andorid support was requested (and there are even open PRs for that), and I didn't manage to tackle this topic and give it the attention that it deserves (sorry about that...).

But a first, wild guess, which I already mentioned in the first answer:

I assume that the background renderer that you showed ist called, rendering the background image. And I assume that after that call, there is a call to GlViewerGles::render. If this is the case, then it will do the glClear call, clearing the previously rendered background image.

A first attempt (quickly, just to see whether this indeed is the issue) would be to remove the glClear call, and see whether this has the desired effect.

(If it does have the desired effect, then one could think about a sensible option to support this feature. This could, for example, be adding some viewer.setClearingBackground(true/false) flag, or making the renderGltfModels method from the base class public, so that it can be called directly, instead of the render method that does the glClear).

[kulnaman]

Admittedly support for android would have been fantastic but I understand its not possible to work on everything of interest.
I did remove the glClear call from the GlViewerGles class but the issue still persisted. In my workflow I am showing the camera screen and on a button click I am simply adding the gltf model to the instance of a GltfViewer class, So whenever I am trying to load the model the screen goes black.
Digging a little bit deeper into the code, I found out that issue is in DefaultRenderedGltfModel.class secifically in the processing of the MeshPrimitiveModel.
Here is the code for the same.

    /**
     * Process the given {@link MeshPrimitiveModel} that was found in a
     * {@link MeshModel} in the given {@link NodeModel}. This will create the
     * rendering commands for rendering the mesh primitive.
     * 
     * @param nodeModel The {@link NodeModel}
     * @param meshModel The {@link MeshModel}
     * @param meshPrimitiveModel The {@link MeshPrimitiveModel}
     */
    private void processMeshPrimitiveModel(NodeModel nodeModel,
        MeshModel meshModel, MeshPrimitiveModel meshPrimitiveModel)
    {
        logger.fine("Processing meshPrimitive...");

        MaterialModel materialModel = meshPrimitiveModel.getMaterialModel();
        TechniqueModel techniqueModel = materialModel.getTechniqueModel();
        ProgramModel programModel = techniqueModel.getProgramModel();

        // Obtain the GL program for the Program of the Technique
        Integer glProgram = gltfRenderData.obtainGlProgram(programModel);
        if (glProgram == null)
        {
            logger.warning("No GL program found for program " + programModel
                + " in technique " + techniqueModel);
            return;
        }

        // Create the vertex array and the attributes for the mesh primitive
        int glVertexArray = glContext.createGlVertexArray();
        gltfRenderData.addGlVertexArray(glVertexArray);
        List<Runnable> attributeUpdateCommands = 
            createAttributes(glVertexArray,
                nodeModel, meshModel, meshPrimitiveModel);

        // Create a list that contains all commands for rendering
        // the given mesh primitive
        List<Runnable> commands = new ArrayList<Runnable>();
        
        // Create the command to enable the program
        commands.add(() -> glContext.useGlProgram(glProgram));
        
        // Create the commands to set the uniforms
        List<Runnable> uniformSettingCommands = 
            createUniformSettingCommands(
                materialModel, nodeModel, glProgram);
        commands.addAll(uniformSettingCommands);

        // Create the commands to set the technique.states and 
        // the technique.states.functions values 
        commands.add(() -> glContext.disable(
            TechniqueStatesModel.getAllStates()));

        TechniqueStatesModel techniqueStatesModel = 
            techniqueModel.getTechniqueStatesModel();
        List<Integer> enabledStates = techniqueStatesModel.getEnable();
        commands.add(() -> {
            glContext.enable(enabledStates);
        });
        TechniqueStatesFunctionsModel techniqueStatesFunctionsModel =
            techniqueStatesModel.getTechniqueStatesFunctionsModel();
        commands.addAll(
            createTechniqueStatesFunctionsSettingCommands(
                glContext, techniqueStatesFunctionsModel));

        commands.addAll(attributeUpdateCommands);
        
        // Create the command for the actual render call
        Runnable renderCommand = 
            createRenderCommand(meshPrimitiveModel, glVertexArray);
        commands.add(renderCommand);
        
        
        // Summarize all commands of this mesh primitive in a single one
        Runnable meshPrimitiveRenderCommand = new Runnable()
        {
            @Override
            public void run()
            {
                //logger.info("Executing " + this);
                for (Runnable command : commands)
                {
                    command.run();
                }
            }
            
            @Override
            public String toString()
            {
                return super.toString(); // XXX TODO
//                return RenderCommandUtils.createInfoString(
//                    gltf, meshPrimitiveName, techniqueId,
//                    uniformSettingCommands);
            }
        };
        
        boolean isOpaque = !enabledStates.contains(GltfConstants.GL_BLEND);
        if (isOpaque)
        {
            opaqueRenderCommands.add(meshPrimitiveRenderCommand);
        }
        else
        {
            transparentRenderCommands.add(meshPrimitiveRenderCommand);
        }

        logger.fine("Processing meshPrimitive DONE");
    }

Can you help me in debugging the above method? I do not understand the inner working of gltf model rendering and any help will be much appreciated. If possible please also point me towards resources where i can learn how the library is rendering the gltf model.

[javagl]

It's not entirely clear why you assume that this is the crucial part of the code here - except, of course, for the fact that this is where all the rendering takes place ;-)

These "render commands" that are assembled and put into a lists there are essentially small Runnable instances that do the low-level GL calls that are required for rendering. For example, referring to the code that you posted, the

commands.add(() -> glContext.useGlProgram(glProgram));

essentially calls

GLES20.glUseProgram(shaderProgramId);

The

commands.addAll(uniformSettingCommands);

calls things like

GLES20.glUniformMatrix4fv(mvpMatrixHandle, 1, false, localMvpMatrix, 0);
GLES20.glUniform1i(textureHandles[0], 0);
GLES20.glUniform1i(textureHandles[1], 1);

And the "main" renderCommand for the mesh primitive calls something like

GLES20.glDrawElements(GLES20.GL_TRIANGLES, INDEX_BUF.length, ...);

And of course, all these calls are done with the specific program, uniforms, and buffers that are required for rendering that particular model.

---

I'm not sure whether that helps you in any way. But I'll try to reproduce the issue (i.e. try to render "my own background", and see whether I can prevent it from becoming black). I cannot promise anything, but will report back later...

[javagl]

OK, from a quick test, I think that the desired effect can be achieved by changing the code at https://github.com/mikikg/AndroidJgltfViewer/blob/master/app/src/main/java/de/javagl/jgltf/viewer/gles/GlViewerGles.java#L147 to

@Override
protected void render()
{
    glDepthMask(true);
    glClear(GL_DEPTH_BUFFER_BIT);
}

to make sure that

• the depth buffer is cleared (so that the model is actually rendered on the screen, and not "hidden" by the "background" due to depth buffer checks)
• the color buffer is not cleared, to keep the previously painted background intact

The result of my test, by loading the "duck" and dragging it around, has a certain aesthetic:

However, if it does not solve your issue, I'll try to render an "actual" backround, by porting the code for the background renderer that you posted.

[kulnaman]

the issue still persists, I am still getting a black screen. Another point, all the commands are regarding model rendering, so why is background getting affected ?

[javagl]

Well, it shouldn't be, obviously - and as it can be seen in the screensot, these commands should not affect the background: The duck is just painted on top of everything.

However: All rendering from JglTF should take place in the `render´ method at https://github.com/mikikg/AndroidJgltfViewer/blob/master/app/src/main/java/de/javagl/jgltf/viewer/gles/GlViewerGles.java#L140 . Right now (mainly because I cannot quickly test or reproduce this), some "bisection steps".

1. If you comment out the contents of this method completely, does it then show the background?
2. If this method does not do any glClear call (but only the renderGltfModels call), does it then show the background?

[javagl]

I'll just close this due to inactivity. The jgltf-viewer packages should be improved and extended, sure, this is on the radar. But there is no direct actionable item for this particular issue.