High value for Window antialiasingLevel causes frequent segfaults on macOS

[saxbophone]

Subject of the issue

When a suitably high value is given for the antialiasingLevel member of ContextSettings, Window objects created with these settings almost always (but not every time) crash the program with a segmentation fault.

Your environment

• OS and version: macOS High Sierra (v10.13.6)
• Version of SFML tested against: v2.5.1
• Built-in Clang: Apple LLVM version 10.0.0 (clang-1000.11.45.5)
• Special compiler flags used: Only -Wl,-rpath and -framework for linking to the SFML libraries.

Steps to reproduce

Compile this code with clang++ sfml_bug_demo.cpp -o sfml-bug-demo -Wl,-rpath,/Library/Frameworks -framework sfml-graphics -framework sfml-window -framework sfml-system:

#include <climits>

#include <SFML/System/Sleep.hpp>
#include <SFML/System/Time.hpp>
#include <SFML/Window.hpp>


int main() {
    sf::ContextSettings settings;
    // on macOS 10.13.6, any value in range 0..(UINT_MAX/2) is fine
    settings.antialiasingLevel = (UINT_MAX / 2) + 1;
    // with value of antialiasing set to (UINT_MAX / 2) + 1, segfault occurs:
    sf::Window(
        sf::VideoMode::getDesktopMode(),
        "SFML Bug demo",
        sf::Style::Default,
        settings
    );
    sf::sleep(sf::seconds(1));
}

Expected behavior

The Window should be created and then immediately disappear (probably with a message on the console indicating that the exact requested settings could not be fulfilled, owing to the ridiculously large value for the antialiasing level).

Actual behavior

Almost every time the program is run, this is the only output on the console:

Error. Unable to find a suitable pixel format.
Segmentation fault: 11

Very rarely, this output appears instead:

Error. Unable to find a suitable pixel format.
Warning: The created OpenGL context does not fully meet the settings that were requested
Requested: version = 1.1 ; depth bits = 0 ; stencil bits = 0 ; AA level = 2147483648 ; core = false ; debug = false ; sRGB = false
Created: version = 0.0 ; depth bits = 0 ; stencil bits = 0 ; AA level = 2147483648 ; core = false ; debug = false ; sRGB = false

Notes

• This same code has been tested on Ubuntu Linux and runs without error.
• The issue is not present with antialiasingLevel being in the range 0..(UINT_MAX / 2), which is 0..2147483647 on my system.
• I realise that it's rather foolish to set antialiasing to such a high value, but there is no documentation specifying any maximum range for the value and it appears that there is something else wrong with SFML that is causing a segmentation fault to happen when given this large value.

[follower]

This might be stating the obvious, but as a random drive-by thought: could something be treating a value as signed int while something else is treating it as unsigned int (or vice versa)? (Thought prompted by relationship between UINT_MAX/2 & INT_MAX.)

The inconsistent nature is a bit of a twist though.

Also does any other OpenGL library have the same issue on the same machine?

[saxbophone]

@follower That is an interesting thought!

For what it's worth, I will add that I did happen to stumble upon the issue originally due to storing a signed integer in an unsigned variable, causing the value to jump to the large one as documented.

I have considered modifying the demo program posted above to use exact literals rather than expressions using the <climits> placeholders, perhaps for explicitness' sake. The thing is, one can't be sure of the width of the built-in integer types in C!

The inconsistent nature is a bit of a twist though.

Yes indeed it is! My guess is it might be a platform-specific implementation detail, perhaps some overlooked constraint in the SFML code which interacts with the macOS-specific OpenGL API...

Also does any other OpenGL library have the same issue on the same machine?

Good question —I'm afraid I'm unable to answer that one though, I do not have any OpenGL programming experience. The only graphics programming I have done is with SDL and SFML. However, from doing a bit of searching within the SFML code, if I'm not mistaken then the ContextSettings class in SFML maps to some kind of equivalent type for OpenGL rendering, so it wouldn't surprise me if it's OpenGL-related.

[saxbophone]

@follower If you or someone else would be able to guide me in creating or supply a minimal test program for verifying if the issue is to do with OpenGL itself, I would be happy to test such a program on my Mac and Linux machines, if it would be useful.

[JonnyPtn]

The joys of unitialised variables...

#1610 will fix the segfault, but to be honest I don't think there's anything else we can do about this issue. It may well be a signed/unsigned conversion somewhere in the apple code, but it's an extreme edge case and the best we could do is just log an extra error saying to use reasonable AA values

[saxbophone]

Hooray! I appreciate that someone took the time to look into it, understand totally that it's an extreme edge-case, that there is now a proposal to add additional code in SFML to handle this gracefully feels like the right way to handle it, thanks!

[JonnyPtn]

I'd say this can probably be closed now

[eXpl0it3r]

This was fixed with #1610

Not sure if it's really worth adding checks for "sane" values.