Anti aliasing is crashing in Dgame 0.6.0-rc.1 on OSX

[JackStouffer]

Continued from #46

Using the following code with 0.6.0-rc.1

import Dgame.Window;
import Dgame.System.StopWatch;

void main() {
    GLContextSettings gl_settings;
    gl_settings.antiAlias = GLContextSettings.AntiAlias.X4;

    Window window = Window(
        1280,
        720,
        "Dungeon Game",
        Window.Style.Default,
        gl_settings
    );
    window.clear();
    wnd.display();
    StopWatch.wait(2000);
}

I get the following result

$ dub
Target derelict-util 2.0.0 is up to date. Use --force to rebuild.
Target derelict-sdl2 1.9.5 is up to date. Use --force to rebuild.
Target derelict-gl3 1.0.13 is up to date. Use --force to rebuild.
Target dgame 0.6.0-rc.1 is up to date. Use --force to rebuild.
Building dungeon ~master configuration "application", build type debug.
Compiling using dmd...
Linking...
Running ./dungeon
SDL version: 2.0.3
OpenGL Context created in OpenGL version 2.1
Set 4 as anti alias level
Error executing command run:
Program exited with code -11

[Dgame]

Did it worked in v0.5?

[JackStouffer]

No, this is the same problem that was mentioned in #46, I am just opening an issue for it now because I can now confirm that the issue is in 0.6 as well.

[Dgame]

I see, I thought that would only occur with Style.AllowHighDPI?

[JackStouffer]

Sorry if I was unclear before, but the issue is with all of the styles, but I have only tested Window.Style.Default in 0.6.

[Dgame]

But if you aren't using anti alias it works? Did you tried gl_settings.profile = GLContextSettings.Profile.Default?

[JackStouffer]

Commenting out the AntiAlias.X4 assignment and adding gl_settings.profile = GLContextSettings.Profile.Default; complies and works. But having both of them, so the code looks like

import Dgame.Window;
import Dgame.System.StopWatch;

void main() {
    GLContextSettings gl_settings;
    gl_settings.antiAlias = GLContextSettings.AntiAlias.X4;
    gl_settings.profile = GLContextSettings.Profile.Default;

    Window window = Window(
        1280,
        720,
        "Dungeon Game",
        Window.Style.Default,
        gl_settings
    );
    window.clear();
    wnd.display();
    StopWatch.wait(2000);
}

gives the same error.

[Dgame]

The only choice to isolate the error would be that you comment out the lines 170, 174 and 182 in Window/Internal/Init.d one after another to see which of them produces the error. Would you do that?

[JackStouffer]

I did that and got this error

$ dub --force                                                                                                                                [13:35:03]
Building derelict-util 2.0.0 configuration "library", build type debug.
Running dmd...
Building derelict-sdl2 1.9.5 configuration "library", build type debug.
Running dmd...
Building derelict-gl3 1.0.13 configuration "library", build type debug.
Running dmd...
Building dgame 0.6.0-rc.1 configuration "lib", build type debug.
Running dmd...
../../../../.dub/packages/dgame-0.6.0-rc.1/source/Dgame/Window/Internal/Init.d(171): Error: undefined identifier result
../../../../.dub/packages/dgame-0.6.0-rc.1/source/Dgame/Window/Internal/Init.d(183): Error: undefined identifier result
FAIL ../../../../.dub/packages/dgame-0.6.0-rc.1/.dub/build/lib-debug-posix.osx-x86_64-dmd_2067-2E5BF8748B103FBA840745DEA4F7B95A/ Dgame staticLibrary
Error executing command run:
dmd failed with exit code 1.

[Dgame]

Yes, commet out the corresponding assert_fmt also. Or introduce a fake int result; to avoid the error.

[JackStouffer]

Alright, I added in the fake result and commented out assert_fmt, and it complied and ran with the following output:

$ dub --force                                                                                                                                [13:49:50]
Building derelict-util 2.0.0 configuration "library", build type debug.
Running dmd...
Building derelict-sdl2 1.9.5 configuration "library", build type debug.
Running dmd...
Building derelict-gl3 1.0.13 configuration "library", build type debug.
Running dmd...
Building dgame 0.6.0-rc.1 configuration "lib", build type debug.
Running dmd...
Building dungeon ~master configuration "application", build type debug.
Compiling using dmd...
Linking...
Running ./dungeon
SDL version: 2.0.3
OpenGL Context created in OpenGL version 2.1
Set 4 as anti alias level
Your anti-alias (4) is too high and will be reduced to 0.
Derelict loaded GL version: 21 (21), available GL version: 2.1 INTEL-10.6.20

[Dgame]

Interesting. Last check, please replace the whole if (gl.antiAlias > 0) { ... }block with this:

if (gl.antiAlias > 0) {
    debug print_fmt("Set %d as anti alias level\n", gl.antiAlias);

    int max_samples;
    glGetIntegerv(GL_MAX_SAMPLES, &max_samples);

    ubyte antiAlias = gl.antiAlias;
    if (antiAlias > max_samples) {
        print_fmt("Your anti-alias (%d) is too high and will be reduced to %d.\n",gl.antiAlias, max_samples);
        antiAlias = cast(ubyte) max_samples;
    }

    if (antiAlias > 0) {
        int result = SDL_GL_SetAttribute(SDL_GL_MULTISAMPLEBUFFERS, 1);
        assert_fmt(result == 0, "Error by initializing OpenGL: %s\n", SDL_GetError());

        result = SDL_GL_SetAttribute(SDL_GL_MULTISAMPLESAMPLES, antiAlias);
        assert_fmt(result == 0, "Error by initializing OpenGL: %s\n", SDL_GetError());
    }
}

[JackStouffer]

Crashed

$ dub --force                                                                                                                                [13:50:16]
Building derelict-util 2.0.0 configuration "library", build type debug.
Running dmd...
Building derelict-sdl2 1.9.5 configuration "library", build type debug.
Running dmd...
Building derelict-gl3 1.0.13 configuration "library", build type debug.
Running dmd...
Building dgame 0.6.0-rc.1 configuration "lib", build type debug.
Running dmd...
Building dungeon ~master configuration "application", build type debug.
Compiling using dmd...
Linking...
Running ./dungeon
SDL version: 2.0.3
OpenGL Context created in OpenGL version 2.1
Set 4 as anti alias level
Error executing command run:
Program exited with code -11

[Dgame]

You're not getting the Your anti-alias (4) is too high and will be reduced to 0. message?

[JackStouffer]

Nope.

[Dgame]

Could you add a print_fmt("Maximal %d samples supported.\n", max_samples);right after glGetIntegerv(GL_MAX_SAMPLES, &max_samples);?

[JackStouffer]

Ok, I am sure that the crash is coming from glGetIntegerv(GL_MAX_SAMPLES, &max_samples); because I inserted print_fmt("Maximal %d samples supported.\n", max_samples); before and after it and it only printed in stdout once

$ dub --force                                                                                                                                [14:04:45]
Building derelict-util 2.0.0 configuration "library", build type debug.
Running dmd...
Building derelict-sdl2 1.9.5 configuration "library", build type debug.
Running dmd...
Building derelict-gl3 1.0.13 configuration "library", build type debug.
Running dmd...
Building dgame 0.6.0-rc.1 configuration "lib", build type debug.
Running dmd...
Building dungeon ~master configuration "application", build type debug.
Compiling using dmd...
Linking...
Running ./dungeon
SDL version: 2.0.3
OpenGL Context created in OpenGL version 2.1
Set 4 as anti alias level
Maximal 0 samples supported.
Error executing command run:
Program exited with code -11

Here is the current code

if (gl.antiAlias > 0) {
        debug print_fmt("Set %d as anti alias level\n", gl.antiAlias);

        int max_samples;
        print_fmt("Maximal %d samples supported.\n", max_samples);
        glGetIntegerv(GL_MAX_SAMPLES, &max_samples);
        print_fmt("Maximal %d samples supported.\n", max_samples);

        ubyte antiAlias = gl.antiAlias;
        if (antiAlias > max_samples) {
            print_fmt("Your anti-alias (%d) is too high and will be reduced to %d.\n",gl.antiAlias, max_samples);
            antiAlias = cast(ubyte) max_samples;
        }

        if (antiAlias > 0) {
            int result = SDL_GL_SetAttribute(SDL_GL_MULTISAMPLEBUFFERS, 1);
            assert_fmt(result == 0, "Error by initializing OpenGL: %s\n", SDL_GetError());

            result = SDL_GL_SetAttribute(SDL_GL_MULTISAMPLESAMPLES, antiAlias);
            assert_fmt(result == 0, "Error by initializing OpenGL: %s\n", SDL_GetError());
        }
    }

[Dgame]

Yeah it seems that you cannot use glGet* without having a valid context. Strange that it works well for other OS. What happens if you erase everything which corresponds to glGetIntegerv(GL_MAX_SAMPLES, &max_samples);?

[JackStouffer]

I changed the code to

    if (gl.antiAlias > 0) {
        debug print_fmt("Set %d as anti alias level\n", gl.antiAlias);

        //int max_samples;
        //print_fmt("Maximal %d samples supported.\n", max_samples);
        //glGetIntegerv(GL_MAX_SAMPLES, &max_samples);
        //print_fmt("Maximal %d samples supported.\n", max_samples);

        ubyte antiAlias = gl.antiAlias;
        //if (antiAlias > max_samples) {
        //    print_fmt("Your anti-alias (%d) is too high and will be reduced to %d.\n",gl.antiAlias, max_samples);
        //    antiAlias = cast(ubyte) max_samples;
        //}

        if (antiAlias > 0) {
            int result = SDL_GL_SetAttribute(SDL_GL_MULTISAMPLEBUFFERS, 1);
            assert_fmt(result == 0, "Error by initializing OpenGL: %s\n", SDL_GetError());

            result = SDL_GL_SetAttribute(SDL_GL_MULTISAMPLESAMPLES, antiAlias);
            assert_fmt(result == 0, "Error by initializing OpenGL: %s\n", SDL_GetError());
        }
    }

and the game complies and runs, but everything still looks aliased and block-y.

[Dgame]

You could try to set glEnable(GL_LINE_SMOOTH); and glHint(GL_LINE_SMOOTH_HINT, GL_NICEST); in your Applikation after you created the Window. And you could try to call int max_samples; glGetIntegerv(GL_MAX_SAMPLES, &max_samples); to see if and how much samples you have. I guess you have 0 and because of that you see no anti alias.

[JackStouffer]

Well I thought that too, but doing

import std.stdio;
import derelict.opengl3.gl;
import Dgame.Window;
import Dgame.System.StopWatch;

void main() {
    GLContextSettings gl_settings;
    gl_settings.antiAlias = GLContextSettings.AntiAlias.X4;
    gl_settings.profile = GLContextSettings.Profile.Default;

    Window window = Window(
        1280,
        720,
        "Dungeon Game",
        Window.Style.Default,
        gl_settings
    );
    window.clear();
    glEnable(GL_LINE_SMOOTH);
    glHint(GL_LINE_SMOOTH_HINT, GL_NICEST);

    int max_samples;
    glGetIntegerv(GL_MAX_SAMPLES, &max_samples);
    writeln("max_samples: ", max_samples);

    wnd.display();
    StopWatch.wait(2000);
}

prints max_samples: 8

[Dgame]

And

glEnable(GL_LINE_SMOOTH);
glHint(GL_LINE_SMOOTH_HINT, GL_NICEST);

does not help?

[JackStouffer]

Not really

[Dgame]

Is this a Texture or a Font?

[JackStouffer]

Font

[JackStouffer]

But after turning the anti aliasing up to 8, there is still some noticeable aliases on the textures as well

[Dgame]

Did you tried the Font.Mode?

[JackStouffer]

The blended mode makes it a little bit better,

but there is some very clear aliasing that should not be seen when using 8x anti aliasing.

[Dgame]

What do you mean with "that should not be seen when using 8x anti aliasing."?

[JackStouffer]

When using 8x on other games that I play, I don't see this kind of artifacting on any of the text or sprites.I see no noticeable difference between the sprites when anti aliasing is on vs when it's off.

[JackStouffer]

Here are two comparison shots:

anti aliasing 0:

anti aliasing 8x:

[Dgame]

Yeah, Anti-Alias is no panacea. ;) It helps to make lines and edges more clear, especially for Shapes, but for Textures it is not that simple, you need more, e.g. a FrameBuffer or Shader (which Dgame 0.6.* includes).

[JackStouffer]

Any pointers on where I can start with FrameBuffers? I can't find anything in the documentation.

[Dgame]

FrameBuffers are not supported from Dgame right now. You need to use plain OpenGL.

[JackStouffer]

Ok, thanks.