New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Zero-copy importer plugin APIs #240

Open
wants to merge 1 commit into
base: master
from

Conversation

1 participant
@mosra
Owner

mosra commented Mar 26, 2018

(I'm overflowing with ideas at the moment and have to prioritize. This lost the idea war.)

The end goal for this:

  • use Utility::Directory::mapRead() to memory-map a huge image / audio / scene file
  • open the memory-mapped file via an importer and parse it without copying the data
  • provide a view on the data to the user (again, no copy)
  • take small slices of the imported data and upload them to the GPU / play them on the soundcard / ..., again no copy
  • have Animation::TrackView instances directly on the memory-mapped data, playing the animation directly off the disk

TODO:

  • Some minimal tests for assertions in the AbstractImporter plugins
  • implement openMemory() as a zero-copy counterpart to openData() in Trade::AbstractImporter
    • delegate one to another based on what features are available
  • implement openMemory() as a zero-copy counterpart to openData() in Audio::AbstractImporter
    • delegate one to another based on what features are available
  • implement openMemory() as a zero-copy counterpart to openData() in Text::AbstractFont
    • delegate one to another based on what features are available
  • redesign Trade::MeshData2D / Trade::MeshData3D to eat an array (view) instead of vectors
    • the whole data being in a single array, particular attributes being aligned views into them
    • support more than just the default type for attributes (in order to avoid pessimistic inflating and make it possible to use e.g. 10-10-10-2 packed quaternions directly)
    • however, for easy usage, allow conversion to default types (reuse the current API returning vectors? or return arrays?)
    • need to deal with BW compatibility somehow (currently returning mutable vectors from accessors, oh well)
  • add an ability to eat an array view to Trade::ImageData2D / ... (convert it to an array with a custom no-op deleter internally)
  • implement Trade::AbstractImporter::mesh2DMemory() / ... that's able to pass through the mapped data in case the file was opened via openMemory()
    • test that it properly asserts when calling this on a file that was not opened via openMemory()
    • test that the returned data views are inside the mapped memory
  • implement Trade::AbstractImporter::image2DMemory() / ... that's able to pass through the mapped data in case the file was opened via openMemory()
    • test that it properly asserts when calling this on a file that was not opened via openMemory()
    • test that the returned data views are inside the mapped memory
  • implement Audio::Importer::data() that's able to pass through the mapped data in case the file was opened via openMemory()
    • test that it properly asserts when calling this on a file that was not opened via openMemory()
    • test that the returned data views are inside the mapped memory
  • Implement this in WavAudioImporter
  • Implement this in FreeTypeFont
  • Implement this in TgaImporter (or not? it flips the image data and converts from ARGB or something ... configuration value?)
  • Implement this in DdsImporter (does it flip the image data?)
  • Optional, for fun: implement ImageView::slice() and then upload a slice to GPU memory
[wip]
[ci skip]

@mosra mosra added this to TODO in Asset management via automation Mar 26, 2018

@mosra mosra referenced this pull request May 20, 2018

Closed

Serious glTF importer bugs #42

5 of 5 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment