Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Zero-copy importer plugin APIs #240

Open
wants to merge 27 commits into
base: master
Choose a base branch
from
Open

[WIP] Zero-copy importer plugin APIs #240

wants to merge 27 commits into from

Conversation

@mosra
Copy link
Owner

@mosra mosra commented Mar 26, 2018

Depends on #371.

The end goal for this:

  • use Utility::Directory::mapRead() to memory-map a huge image / audio / scene file
  • open the memory-mapped file via an importer and parse it without copying the data
  • provide a view on the data to the user (again, no copy)
  • take small slices of the imported data and upload them to the GPU / play them on the soundcard / ..., again no copy
  • have AnimationData, MeshData, ImageData work directly on the memory-mapped data

TODO:

  • Some minimal tests for assertions in the AbstractImporter plugins
  • implement openMemory() as a zero-copy counterpart to openData() in Trade::AbstractImporter in progress
    • delegate to openData() if feature not supported
  • implement openMemory() as a zero-copy counterpart to openData() in Audio::AbstractImporter
    • delegate to openData() if feature not supported
  • implement openMemory() as a zero-copy counterpart to openData() in Text::AbstractFont
    • delegate to openData() if feature not supported
  • redesign Trade::MeshData2D / Trade::MeshData3D to eat an array (view) instead of vectors -- done in #371 instead
  • add an ability to put an array view to Trade::ImageData2D / ... (convert it to an array with a custom no-op deleter internally) -- done in #371 instead
  • implement Trade::AbstractImporter::mesh2DMemory() / ... that's able to pass through the mapped data in case the file was opened via openMemory() done differently in #371
  • implement Trade::AbstractImporter::image2DMemory() / ... that's able to pass through the mapped data in case the file was opened via openMemory() done differently in #371
  • implement Audio::Importer::data() that's able to pass through the mapped data in case the file was opened via openMemory()
  • Implement this in WavAudioImporter
  • Implement this in FreeTypeFont
  • Implement this in TgaImporter (or not? it flips the image data and converts from ARGB or something ... configuration value?)
  • Implement this in DdsImporter
    • We need some "don't flip Y" flag
  • Implement this in StanfordImporter in progress
  • Implement this in TinyGltfImageImporter
  • Optional, for fun: implement ImageView::slice() and then upload a slice to GPU memory
@mosra mosra added this to TODO in Asset management via automation Mar 26, 2018
@mosra mosra mentioned this pull request May 20, 2018
5 of 5 tasks complete
@mosra mosra mentioned this pull request Sep 13, 2019
70 of 70 tasks complete
mosra added 26 commits Nov 5, 2019
Better for checking accidents, as picking a wrong primitive / index type
can lead to *serious* rendering issues. Similarly to a change done to
(Compressed)PixelFormat in 2019.10.
Not yet sure if I should follow the PixelFormat naming or not. Maybe
will change this later.
Otherwise they take up too much space.
With API analogous to the (relatively) new AnimationData -- with one
buffer containing all index data and one buffer containing all vertex
data, both meant to be uploaded as-is to the GPU.

This will eventually replace MeshData2D and MeshData3D, backwards
compatibility and wiring up to other APIs will be done in follow-up
commits.
Deprecating of the old ones comes later.
Will be used for MeshData that don't own the memory.
Turns out the design wasn't so simple after all. AnimationData and
ImageData classes will follow with similar changes.
Follows the change done in MeshData.
This was designed mainly to avoid one extra allocation in the MeshData
-> MeshData2D/3D backwards compatibility conversion constructors, but I
think it could have uses outside of there as well -- sometimes you just
don't want to branch around all possible underlying types when working
on vertex data.
So users aren't force to specify everything on their own. It makes the
test code a bit less painful. But just a bit.
Easier to write. Need to take extra care with default deleters.
First step towards an ability to expose data compiled into an executable
through MeshData without having to allocate anything.
This makes it possible to have fully allocation-less MeshData, with
statically defined indices and attributes. Only the final MeshData
construction needs to be done at runtime because Array is not constexpr,
but that isn't anything heavy anyway.
For backwards compatibility these will delegate to the new MeshData
interfaces for 3D (and nothing for 2D, because so far there were no 2D
scene importers).
There's a ton of parameters and it's just unreadable without.
TODO: normal generation (needs interleave() to operate on MeshData
  directly, and a way to make room for the normal data there)
Just two primitives converted and it's already ~200 kB saved in Debug?
Wha?

TODO: do I need to update docs (types, attribs)? what about
  backwards-compat includes?
@mosra mosra force-pushed the meshdata branch 3 times, most recently from bf0963b to 1c8d552 Feb 22, 2020
@mosra mosra force-pushed the meshdata branch 16 times, most recently from 53f96e0 to eaa59cc Mar 2, 2020
@mosra mosra force-pushed the meshdata branch 2 times, most recently from 6f62217 to a4bf0e6 Mar 10, 2020
@mosra mosra closed this Mar 11, 2020
Asset management automation moved this from TODO to Done Mar 11, 2020
@mosra mosra reopened this Mar 11, 2020
Asset management automation moved this from Done to In progress Mar 11, 2020
@mosra mosra changed the base branch from meshdata to master Mar 11, 2020
@mosra mosra modified the milestones: 2020.0a, 2020.0b Jun 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Asset management
  
In progress
Linked issues

Successfully merging this pull request may close these issues.

None yet

1 participant