2023.0a release

[mosra]

Opening early mainly to avoid forgetting the tiny but important things to do (in addition to all issues/PRs that are in this milestone). Goddamit this is growing fast.

• MSVC 2022 compatibility, drop 2017 ES from the CI to make room -- Update compiler workarounds for MSVC 2022 corrade#124
• Integrate https://git.struna.me/skrat/magnum-boostrap-qtquick-min
• Integrate https://github.com/alkavan/magnum-rpm-specs-f28 and/or RPM build #537 and related PRs
• EmscriptenApplication doesn't report touch drag as move events (Sdl2Application did, I think), fix
  • It should also implement some native touch events
• update reST docs in bindings to use explicit :ref:, backticks somehow don't work anymore -- mosra/magnum-bindings@be5ffdd
• Bind only the first attribute of given type in MeshTools::compile() -- if a mesh has two texcoord sets, both get bound, which basically means the first get overwritten by the second one and that's not correct -- c357a44
• ⚠️ Assert redirection to default output makes the messages invisible on Android apps (basically need a variant of https://gist.github.com/mosra/566aedd25eadd07927d4f7e385aac931, but for Android apps only)
• Enable dart in magnum-integration vcpkg package (uncomment, change dart to dartsim and enable /permissive- on msvc -- https://gitter.im/mosra/magnum/archives/2020/09/30)
• With mesh = GL::Mesh{NoCreate}, GL::Context::current() is still accessed on mesh destruction (needs to drop/rework the vector-based internals and turn them into a type-erased Array with a growable deleter of a concrete type, which will avoid the need to ask state tracker which code path to use) -- f7a6d79
• EXT/OES_draw_elements_base_vertex is broken (no-op?) on ANGLE multidraw doesn't work on ANGLE due to EXT_multi_draw_elements not implemented, use the ANGLE ext there instead - 5f184bb
• Add an option for Corrade resources to be aligned (and null-terminated), needed by @Squareys -- mosra/corrade@9ef4b4a
  • also investigate corrade_add_resource slow compile corrade#94 (adding an explicit size could improve things a bit at least)
• Prefix all CMake options with CORRADE_ / MAGNUM_ to make things less shitty for subprojects, provide some backwards compatibility to avoid breaking everything -- mosra/corrade@878624a, b983781 and similar commits in other repos
• Check for src == dst in Directory::copy() (needs to compare inodes to be robust) @Squareys
• Fix a bug in sphere/frustum intersection algo Math: Fix Math::Intersection::sphereFrustum #482
• uploading 512x512 tiles of a 4096x4096 CompressedImage3D (4x4, 128bit blocks) doesn't work correctly, data size is calculated as 2034 kB instead of 20148 for some reason -- investigate
  • note to self: I have a stash of this + various other fixes, need better tests to catch this properly
• make FreeTypeFont library thread-local on windows (it needs to be exported so harfbuzz can use it, but dllexported symbols can't be threadlocal because WINDOWS GODDAMIT WINDOWS)
  • The "global" state also breaks/asserts when the plugin is loaded through multiple plugin managers at the same time, what to do?
  • Compile Windows FreeType binary on https://github.com/mosra/magnum-ci so it's tested there -- ci: Add win64 build of freetype magnum-ci#6, mosra/magnum-plugins@37a1866
• Look into mitigating crashes in UASTC-enabled Basis so we can upgrade -- Upgrade Basis plugins to 1.15 magnum-plugins#112
• Add an ability to iterate keys in ConfigurationGroup -- mosra/corrade@cbbf926
  • implement option propagation in Any plugins and avoid specifying --importer every time I need to set some plugin options -- e6d673c
• Add -I / -C shortcut options for --importer / --converter to all tools -- 6be0248
• allow omiting the Importer / SceneConverter / ImageConverter suffixes
• Look into website caching (cache-busting URLs?)
• Finally deshittify Doxygen URLs (should also save quite a bit on the search data size)
  • Source links
  • List of all members including inherited members (add those to search also)
• Improve docs:
  • Mark SceneGraph, ResourceManager and related APIs as being in maintenance mode from now on
  • ResourceManager, Resource (what is T and U, what is "resource state"... basically throw away everything existing and start from scratch)
  • "Light drawing" example in Drawable docs (based on what I have in magnum-player) -- or maybe do that directly in the viewer example once SceneData is in?
  • Mention the difference between Matrix3x3 and Matrix3 more prominently (in the type overview, in Matrix3 docs...) and for Matrix4 as well -- 3097c4c
  • Simpler 1D intro docs in StridedArrayView with example snippets and corresponding output, take the important stuff (equations?) from https://blog.magnum.graphics/backstage/multidimensional-strided-array-views/
  • Document basic usage of AbstractImageConverter / AbstractSceneConverter classes -- 4a06f4d
  • Document what's the iOS default framebuffer and why also a default renderbuffer needs to be taken care of -- https://gitter.im/mosra/magnum/archives/2021/10/14?at=61686ead8c019f0d0b3563d9
  • Document use of ANGLE on Mac, stop saying it's Windows-only in progress -- 4be0f6d
  • Improve GL::Texture documentation, show a basic single-mip case and then either uploading all mips (mention file formats that know it) or generating them; reference the example
• Switch to the new NDK layout, update docs about LIB_SUFFIX autodetect LIB_SUFFIX on new NDK versions, mention CMake 3.20 and an explicit FIND_ROOT_PATH on older versions -- bc0bb07
• Something is fishy with the Area Lights example
• Add a Mac + ANGLE build, use a binary from Mac Angle magnum-ci#3 actually running against SwiftShader would be better (and using it for Vulkan as well) -- 031d1ce
• Tell travis to fuck off Port to travis-ci.com so we have usable PR reports again actually, yes, tell Travis to fuck off
  • Port remaining repos to CircleCI (integration, examples, extras, bindings, bootstrap, flextgl left)
  • What about Android and iOS builds? I really don't want to rework these as they rely on some ancient infrastructure in order to actually run the tests (hopefully maybe Travis actually responds to that e-mail at some point?) Restore Android and iOS builds on another CI service #523
  • needs digging into the .com API in order to port https://magnum.graphics/build-status/ (last time I checked it was HIGHLY NONTRIVIAL)
  • Same for codecov (@codecov-commenter comments but complains we should be using the GitHub App action thingy whatever)
• Switch all SceneGraph examples to use addChild() / addFeature() instead of naked new -- the original reason for this "being based off Qt" makes no sense nowadays (not that many people use or know Qt anyway) and feels strange, especially in cases when the expression result isn't used anywhere nope, this lib is in maintenance mode now
• tangent computation based on http://www.mikktspace.com/
  • add an option to compile() no, deprecate that and turn it into standalone tools instead
  • expose together with normal computation in magnum-sceneconverter
• expose blend function / equation in python bindings -- python: expose GL::Renderer::setBlendEquation(), GL::Renderer::setBlendFunction() and related enums. magnum-bindings#9
• ability to set window position in Configuration for GLFW and SDL (and later as well), figure out what to do on HiDPI and multi-mon systems
• String[View] APIs returning a view should not be a noun because that sounds like the original object is modified (so no stripWhatever() but rather exceptWhatever()) -- mosra/corrade@b310615
  • what about join(), split() and such? this is fine, as it kinda implies returning a result
  • rename ArrayView::except() to exceptSuffix(), do the same elsewhere and in StringView, and also suffix() to exceptPrefix(), with suffix() reserved for the future to take N chars at the end -- mosra/corrade@c6f3ec5
• Introduce ArrayView::isEmpty() and similarly for others, deprecate empty() -- mosra/corrade@6bebd0e
• Deprecate Utility::Resource::get() in favor of getString() returning a StringView so we can repurpose get() to return an ArrayView and drop getRaw() -- mosra/corrade@64b922f
• decide if plugin options should be camelCase or snail-case (former is consistent with existing options, latter looks better on the command-line -- magnum-shaderconverter -fmax-struct-members=3 --validate a.spv) -- decided to stay with camelCase because it's faster to type, dropping the -f option as it's redundant
• EmscriptenApplication doesn't implement Module.doNotCaptureKeyboard
• Convert WebApplication.css to use classes instead of IDs (also, prefix with .magnum- or some such?) EmscriptenApplication: custom canvas id, multiple applications on one page #480
• Download the Emscripten binary after setting Module properties EmscriptenApplication: custom canvas id, multiple applications on one page #480 (comment)
• Port remaining Find modules to if(IN_LIST) for component classification (integration, extras, bindings left)
• Implement REASON_FAILURE_MESSAGE in remaining Find modules (integration, extras, bindings left)
• Investigate whether using normal matrix to transform tangents is really okay or not (probably not, source)
• Find module for SPIRV-Tools is broken with latest releases Use Homebrew's std_cmake_args homebrew-magnum#6 -- mosra/magnum-plugins@d8d016e
• dynCall() not found with Emscripten 2.0.11 (happens when drawing anything)?? -- e65d6cf
  • they're reworking it somehow, adapt to how they fixed it
  • also it seems like the fix broke the idling? ugh
• Matrix[34]::from(scale, rotation, translation) in addition to the one combining just rotation and translation
• AlwaysOnTop flag for Application classes (huh, already done for 2020.06, why this item is even here?!)
  • probably also needs some way to undo this (WindowFlag::AlwaysOnTop has no effect on Windows #536)
• write my own Containers::Pair because std::pair isn't trivially copyable and so Algorithms::copy() doesn't work on it, ffs (https://danlark.org/2020/04/13/why-is-stdpair-broken/) -- mosra/corrade@e9d43fa
  • another reason is that it performs a copy on construction where it shouldn't (where was that code? update it to use Pair) -- 7f4500d
• ⚠️ fix the flextGL symbols not being shared across SOs, but adding one more pointer indirection is probably not wise for perf so it needs some other solution
  • could GL::Context remember the context handle it was created with, cache it in the struct and then if not the same when switching / when current() is queried, copy the data over? it won't handle all cases tho, GL symbols can be accessed even without GL::Context involvement :(
• ⚠️ symbol deduplication doesn't work with Python bindings because Python loads it with RTLD_LOCAL (but this has to be recent, also the bindings warn that X11 symbols are not found, i don't remember this before)
  • workaround with an explicit sys.setdlopenflags(flags | ctypes.RTLD_GLOBAL) before importing magnum -- mosra/magnum-bindings@b4d437b
  • there's some STB_GNU_UNIQUE that's for the same thing? https://stackoverflow.com/questions/50402314/static-link-stdc-without-stb-gnu-unique-cause-memory-leak-when-dlclose
• vcpkg should delete the plugin lib directory only on windows (https://github.com/microsoft/vcpkg/blob/b759049a36728d18260963799a56e6b19cb4a2ef/ports/magnum-plugins/portfile.cmake#L102-L103, same for magnum) -- Trouble loading plugins in Viewer demo magnum-examples#96, [magnum,-plugins] Fix deployment of plugin types and incorrect removal of plugin lib dir on unix microsoft/vcpkg#16245
  • https://github.com/microsoft/vcpkg/blob/master/ports/magnum/magnumdeploy.ps1 should take care of the new scene and shader converters as well -- [magnum,-plugins] Fix deployment of plugin types and incorrect removal of plugin lib dir on unix microsoft/vcpkg#16245
• installing magnum-plugins triggers the installation of all the features in magnum -- [magnum/magnum-plugins] fix magnum-plugins building all magnum dependencies microsoft/vcpkg#16657
• bundle CMake modules so depending projects don't need to copy them (Figure out how to bundle 3rd party Find modules #436, possibly fixes [magnum/magnum-plugins] fix magnum-plugins building all magnum dependencies microsoft/vcpkg#16657 as well)
• vcpkg installs sceneconverters to a wrong location on linux?! vcpkg/installed/x64-linux/lib/libMeshOptimizerSceneConverter.a this is the desired behavior actually
• Assimp built by hand on macOS has broken cmake config and FindAssimp doesn't work for it since mosra/magnum-plugins@17561c8 -- mosra/magnum-plugins@c7aa467
• from is a keyword in python and so having any python bindings with this name is stupid -- mosra/magnum-bindings@cbb5358
  • new pybind has some shittiness in printing enum values (magnum.shaders.Flat2D.<Flags.NONE: 0>, thefuck?!), affecting doc builds -- mosra/m.css@c6707e1
  • python bindings gl.Context.current() causes an exception (no current context) when generating docs, patch to be a noop in conf.py
• Change extension-dependent GL functionality to be non-member funcs (so we don't need to do the includes for MSVC) and referencing directly the target GL API where possible to avoid extra indirections (uniforms, drawinstanced, KHR_debug APIs etc) in progress
• Intel-specific workaround for shader output logs added back in 3bb62b7 might be obsolete already, remove if not reproducible anymore even on older systems (not reproducible on 27.20, neither 22.20)
  • OTOH WebGL probably needs the ANGLE workaround as well (superfluous newline) no, logs are clean
• Add addFlags() / clearFlags() to AbstractImporter, AbstractImageConverter... APIs and encourage their use in order to prevent issues when FlipY gets added implicitly for images -- 29ac7b5
• ability to specify NoError in all apps, --magnum-gpu-validation no-error as well -- 24138e2, 45ccf50
• Homebrew packages fail to build if binutils is installed -- reproduce, fix: https://gitter.im/mosra/magnum/archives/2021/03/04?at=6040d89fe8267a46f2efe1e6
• Move Quaternion::toEuler() to a Matrix4, since that's where it should live, add other variants of it (ZXY, ZYX, ...) and ensure they're really in the order that is advertised (e.g. this one is labelled as ZYX in threejs but it's actually XYZ)
• Implement std::string_view compat for StringView, Debug and format() -- mosra/corrade@cf0bd1f, mosra/corrade@e7ec9c5, mosra/corrade@46f74dc, mosra/corrade@1ce936c
• Move Containers/Tags.h to root namespace as SIMD stuff will live there as well and we then don't need to put weird aliases to Magnum -- mosra/corrade@c0267de
• glslang 11.2.0 in homebrew causes unresolved symbol errors when loading the plugin? -- mosra/magnum-plugins@e8ebb74
• SceneConverter::convertToFile() needs to be changed to (from, to), all others have it like this -- 06e039d
• scene, image, shader and font converter plugins need to return Optional<Array> instead of Array consistently with Utility::Path API changes, otherwise it doesn't enforce proper error handling -- 51f7862 (font converters have a TODO for later)
• Adapt to changes in meshoptimizer 0.16 -- mosra/magnum-plugins@358f0dd
• Phong specular color should have a Color3 setter that uses alpha=0 (the default conversion to Color4 will set it to 1, breaking alpha-blended materials)
  • Also the specular with alpha=1 probably doesn't really do the "soap bubble" effect at all? I suppose I don't even have a test for that.
• Implement KTX2 support -- Trade: KTX importer magnum-plugins#103
  • and proper Basis-enabled glTFs -- Ktximporter: forward Basis Universal compressed images to BasisImporter magnum-plugins#113
• opt out of FLOC
• MeshTools::generate{Flat,Smooth}Normals() operating directly on MeshData, because it's too inconvenient to not have it
  • deprecate in compile()
  • also for flipFaceWindingInPlace() and flipNormalsInPlace()
  • and expose in magnum-sceneconverter
• MeshTools::generateTrivialIndices() that outputs 0, 1, 2, 3...
• Arbitrary insertion/deletion for growable arrays, finally -- mosra/corrade@bcdbde2, mosra/corrade@6812a28
  • also arrayRemoveUnordered() that swaps the hole with the last value instead of moving everything after for containers that don't need to preserve order -- mosra/corrade@c9089f7
  • should I go with arrayRemoveAt() for removing by index and reserve arrayRemove() / arrayRemoveAll() for removing by value? arrayRemove() is shorter and thus should do the faster operation, go with arrayRemoveValue() for a version that performs a lookup first?
• Add arrayClear()
• Finalize block-size-aware CompressedImage support
  • blocks DDS and KTX2 import/export (Trade: KTX importer magnum-plugins#103)
  • Introduce ValueInit / NoInit [Compressed]Image constructors that allocate
• Add Trade::AnimationTrackTargetType::ImageLayer for animated image support, and possibly others to prepare for the new glTF extension
  • Convert StbImageImporter to use this
  • Implement in KTX import / export -- Trade: KTX importer magnum-plugins#103 (comment)
• Ability to reference a morph target (a pair of mesh ID + morph target ID inside that mesh) in AnimationTrack
  • Add a metadata field with morph target ID to MeshAttributeData
• Make AnimationData constructors less annoying ("backport" the newly learned usability features from MeshData and SceneData) -- 94e9961
• Add array Trade::TextureType values -- 4c4b259
• Add Cube / Array flags to all image classes, adapt Basis, DDS, KTX and EXR plugins to use them (details in Trade: KTX importer magnum-plugins#103) -- f14e15c
  • Add a converter-/importer-specific flags to override those? (e.g., open a (misinterpreted) 3D image as an array, force 3D image export to be understood as a cubemap...)
  • Also YDown / YUp, ZForward / ZBackward, for X as well, with both being set meaning the GL variant and none meaning "any"; enforce in GL upload functions
    • needs flipping of generic block formats implemented so we don't assert with no way to fix that -- 3ccb7a9
    • also a way to set desired orientation on import (not on export tho, there it either embeds the metadata or flips to match the file format implicit orientation)
• Check that T in image.pixels<T>() actually matches the format to avoid accidentally interpreting the data as some completely different type that only coincidentally happens to have the same size and thus passes the current assertion
  • keep the size check for implementation-specific format
  • would require a dependency on PixelFormat.h, not nice :/
• Add an ability to supply a list of ImageViews to AbstractImageConverter to support mip level export (blocks Trade: KTX importer magnum-plugins#103) -- 11268c6
  • Adapt BasisImageConverter for this -- Upgrade Basis plugins to 1.15 magnum-plugins#112
  • Adapt OpenExrImageConverter + OpenExrImporter for this -- mosra/magnum-plugins@21698e9
• Implement & test 1D, 3D and compressed image export in AnyImageConverter, use the KTX plugin for testing -- a130f26, 876976b, 9fbc637, 1f33071
• Mutable access to MaterialData values (images, meshes, ... all have this already, so why not materials) -- 78cf81b
• glTF Vertex color import broken AssimpImporter their bug (floats get extracted as shorts??), I can't fix the world
• ⚠️ Change "$Layer" in MaterialData to " Layer" to avoid conflicts with attributes starting with $, whitelist a whitespace for just this case alone (unblocks AssimpImporter: support PBR magnum-plugins#91) -- aeeff73
• Color3::fromInt() that doesn't go through sRGB decoding, make it named weirdly so it's not accidentally used in sRGB-correct workflows (fromLinearInt()?) -- 4a87a0b
  • and finally Color3::from[Srgb]HexString(Containers::StringView) (name?), implement it in a cpp (for all types??) to avoid the StringView include
  • what about to[Srgb]HexString()? returning a String? ugh actually why not, it's SSO'd
• make plugin interface strings picked up automatically from a header so i don't need to manually update them every time -- 12fc59b
• Internal assertion in Corrade's arg parser when a positional argument is specified as named -- mosra/corrade@6455c5a
• ⚠️ AssimpImporter 5.1.4 asserting when opening CesiumMilkTruck.glb
• Mention new projects in release announcement:
  • BonViewer?
  • Mechanica?
  • light.vn
  • Monopticon (or did I mention it already?)
  • https://github.com/bfeldpw/battlesub and the other project
  • https://github.com/pezcode/mosaiikki
  • https://twitter.com/holllyhook/status/1299249703868137472
  • thesis from Janos -- https://github.com/Janos95/Phasefields
  • https://github.com/TheOpenSpaceProgram/osp-magnum (ask the authors!)
  • https://visualcomputing.ist.ac.at/publications/2021/MADYPG/
  • https://rat-gui.ch/index.html, Magnum-based GUI for https://gitlab.com/Project-Rat/rat-models/-/tree/dev
  • https://github.com/Auburn/FastNoise2
  • https://nimbro.net/AVATAR/, https://arxiv.org/abs/2109.11373, Magnum used for for low-latency VR rendering of the robot video stream
  • https://github.com/kualta/core
  • https://github.com/FonzTech/BreakMyCircle
• CircleCI deprecates 16.04, upgrade to 18.04
• Finish PluginManager and GL string cleanup (Bye bye std::string in the GL library #499)
  • ideally also Arguments, Configuration and Utility::String
• [compressed]PixelFormatName(), vertexFormatName(), meshIndexTypeName(), *ForName() in importer APIs for better support of custom formats
• Update to Vulkan 1.3 (expose new feature bits, etc.)
• Vulkan swapchain support
• Vulkan builtin shaders
  • Needs SPIR-V patching to clean up inputs that are used only by code disabled via a specialization constant
  • Needs convenience APIs for binding buffers & textures
• Add a non-hacky way to disable dynamic plugin lookup in PluginManager (currently it's done by passing "nonexistent" to the constructor, which is shitty)
• copysign(), signbit() scalar/vector utils in Math
• Add GL suffix to Text::GlyphCache, Text::DistanceFieldGlyphCache and TextureTools::DistanceField
• All debug-only assertions in Math should be (non-debug-only) exceptions in Python, the extra overhead doesn't matter there and it's too error-prone without
• Update Vcpkg packages
  • The root cause for Corrade fails to compile with emscripten 3.1.22 corrade#153 (ANSI_COLORS should be just on Windows)
  • New plugins (GlslangShaderConverter, SpirvToolsShaderConverter, OpenExrImporter, OpenExrImageConverter, GltfImporter, GltfSceneConverter, StbResizeImageConverter, StbDxtImageConverter, AstcImporter, SpngImporter, UfbxImporter, BcDecImageConverter, EtcDecImageConverter, ...)
  • New libraries (SceneTools, ShaderTools, MaterialTools, others?)
  • New tools (magnum-shaderconverter, others?)
  • Prefix CMake options
  • EmscriptenApplication on Emscripten instead of SDL2: Update vcpkg packages to include EmscriptenApplication on Emscripten #607

[ttnghia]

More examples please 😋

[mosra]

@ttnghia yeah I know, sorry that I didn't merge your latest one yet 🙈

[ttnghia]

No problem 😄 Once I'm done with my job search (in another month or so), I may have more examples for you 😏

[jonesmz]

Were there plans to make a release in 2021 still?

I'm not in any particular need of one, I just prefer to use official release when possible instead of some random commit (e.g. the sdl2 fix you helped me with earlier this year)

[mosra]

I'd like to make one, yes. The reason why there wasn't any release since last year is that I'm unfortunately still busy on really huge features and I'd like to get them into a somewhat presentable state first, to have something to talk about in the release announcement. Those are:

• the low-level Vulkan backend, which is missing SDL/GLFW integration and some shader-related bits
• the new material import APIs, which support custom material properties but neither Assimp nor glTF importers make use of them yet, making it all kinda useless
• the (in-progress) data-oriented scene import (SceneData rework #525), which finally plugs all holes of the legacy import pipeline and allows Vulkan-friendly batch rendering straight from the imported data

Since a release is mainly a marketing booster for the project, announcing a new version with "look, there's a fancy new <thing> but please ignore it, it's not worth talking about yet" would be wasted effort :) I get your point tho, and sorry that it drags on for so long, fortunately Git master is always stable so relying on some random commit isn't a dangerous thing to do.

[jonesmz]

Understood!

Thanks for the explanation.

I'm not in a rush, so no big deal to me.

[EhWhoAmI]

How much closer are you to a release?

[mosra]

Of the three main tasks I mentioned in a comment above, it's the Vulkan stuff remaining. Plus the thousand tickboxes in the issue tasklist.

[EpsilonKu]

Happy new year!!! Time to change title, lmao