Open Voxel File Format Spec ideas

[dougbinks]

A short discussion in the Vengi discord led to me mentioning on the VoxelGameDev subreddit that:

I'm hoping to create an open Voxel interchange/transfer format. My thoughts are that I would make this based on gltf's json format for everything but the raw voxel data, which would likely be in RLE encoded binary arrays. It would be great if it could support different bits per voxel (at least 8 and 16) and perhaps other data than just material indices (Avoyd has 8bit density), but that would be a stretch goal.

If anyone's interested in following along with this let me know. I'm thinking of starting by making a spec sheet on Github, and will notify any folk who reply when I get around to putting the first draft up.

Essentially a number of tools such as Vengi and my own Avoyd have outgrown the MagicaVoxel .vox format, and we need something else. This is the place for this discussion and perhaps an initial implementation.

[dougbinks]

My thoughts on requirements (not set in stone):

1. As simple as possible - implementations should be able to be lightweight
2. Open and extensible
3. Arbitrary transforms
4. PBR material properties including those needed for transmissive volumes
5. Indexed material voxels
6. No limitation on size of voxel volumes
7. Stretch goal: other per voxel properties
8. Animation
9. Relatively efficient but simple storage and serialization performance

[JuanDiegoMontoya]

As a game dev (not someone making editors or sims), I agree with these requirements. I'd like to expand on some of these points to add what I'd like to see specifically:

3. Like glTF, a scene should consist of a node hierarchy, where nodes can optionally point to voxels.
4. Agree, though see my next point.
5. How should voxels map to materials? Is a voxel just a material index? What about physics sims or other situations in which it's more efficient to store data (such as velocity or density) directly in the voxels themselves?
   a. Voxel games render voxels in a variety of ways, from simple cubes to isosurfaces to little meshes (e.g. Minecraft fence) or even small blocks (my game does this). Do we care to support these? I would argue that it is important to have similar visual fidelity between applications, but it could be difficult to support more than a basic rendering scheme, in which case it may be better to not embed this in the format, or to leave it as something to put in an extension.
6. I agree and think that both sparse and dense voxel representations should be supported as neither one is clearly more optimal than the other in most situations.

I'll add that it'd be nice to make as many aspects of this format ignorable as possible, to simplify the process of writing an importer or exporter for an application that only cares about a subset of the features. For example, my game uses .vox files to represent block materials. In this application, I only care about a single volume, its size, and the material palette, but not the transform hierarchy.

[dougbinks]

5. How should voxels map to materials? As the conversation with @mgerhardy below I think my stretch goal 7 is probably a requirement, i.e. we may need a set of extendable attributes and types similar to how glTF manages mesh data.

For your block materials .vox might be the simplest option, but just loading the first models of this format without their scene (as you likely do with .vox) should be made pretty simple by design.

[dougbinks]

My thoughts on potential implementations based on other file formats (an edit of what I posted on r/VoxelGameDev)

The format is intended for complete scenes as well as models.

There are two main transfer formats in the gamedev space, glTF and Universal Scene Description (USD), and for Voxel formats the MagicaVoxel .vox format.

MagicaVoxel .vox

This is a relatively simple format which is extensible, however in my view it does not meet the requirements I have:

1. Proprietary: ephtracy could change the format in a way which breaks our extensions
2. Scene hierarchy has an implicit pivot at the centre of the voxel bounds
3. Limit on size of voxel volumes as encoded as 8bit positions and 8bit material id
4. No arbitrary transforms
5. Material properties are somewhat esoteric
6. Limited material palette (only 256 entries)
7. Animation is integer frame based

glTF

glTF is fairly simple. It has some issues dealing with very large scenes (the binary .glb format is limited to 32bit size), but it is extensible so this could be worked around or external assets used. There's a fair amount of tooling around glTF, and hierarchies plus animations should work with this tooling even if they can't load the voxel data extension components.

glTF does not have any volume description, however it has extensions for volumetric parameters which are applied to the interior of meshes such as index of refraction and transmission/attenuation. We can reuse these, and add extensions for other parameters (scattering etc.).

USD

USD is more complicated, but more powerful.

USD has volumes usdVol which are usually represented by an OpenVDB asset. Technically with USD one could likely throw a load of open source code at the problem and with a 40MB library (compiled) solve the problem without any further work than specifying how to handle indexed OpenVDB materials (OpenVDB can handle integer voxels).

Conclusion

I personally favour lightweight code, and thus glTF. There are simple C and C++ libraries for using it, and the base information is very simple JSON.

On the other hand USD is far more powerful and already has volumes.

Your thoughts are welcome below

[mgerhardy]

it would be nice to also support RGBA as voxel data and omit the palette. There are formats out there that are storing RGBA values already. Everything in Qubicle for example (https://gist.github.com/tostc/7f049207a2e5a7ccb714499702b5e2fd). So I would say, 32bits and 24bits are also a must have if you don't want to lose data when converting from one format into another.

[dougbinks]

Agreed, that would come under Stretch goal: other per voxel properties so I would guess we might need to move that to an actual goal?

I was considering using something similar to glTF's attributes (which are like OpenGL/Vulkan ones), though I'm unsure if we want the full attribute->accessor->bufferView indirection.
https://github.com/KhronosGroup/glTF-Sample-Models/blob/main/2.0/Box%20With%20Spaces/glTF/Box%20With%20Spaces.gltf#L51-L54

We certainly want the ability to have a use and a type, i.e.:
Usage, number, type
RGBA_COLOR, 4, UNSIGNED_BYTE
MATERIAL_INDEX, 1, UNSIGNED_BYTE
MATERIAL_INDEX, 1, GL_UNSIGNED_SHORT

[Longor1996]

Neat! Let me just quickly dump a few (very shortened) points from my previous notes on this idea...

EDIT: Copying without consistency/duplicate checking t'was a bad idea; see replies.

---

• ...
• Mutability / editing enforces some database-ish properties, exposing issues like integrity.
• Efficient indexing and access of subvolumes (ie: non-finite/too-big-for-RAM volumes).
  • How do you define a volume index? It almost directly depends on the defined storage hierarchy.
  • This makes glTF by itself non-viable, as it can't efficiently store a large index.
  • ...
• Paletted and non-paletted storages (ie: "tile entities", bags-of-properties).
• Non-discrete voxels (ie: isosurfaces, transvoxel, dual contouring)
• Multiple layers (ie: think fluids, wires, covers, etc) and thus multiple palettes.
• Temporal Dimensions (ie: history of changes and/or replays).
• Alternate Dimensions (ie: overlapping worlds, possibly scaled, like Overworld/Nether in MC)
• Higher Dimensions (ie: literally 4/5d).
• Holes & Sparsity (ie: infinite world generation).
• Incompleteness (ie: phased world generation).
• Arbitrary spatial depth/scale (ie: octrees).
• Different data layouts over depth/location.
• Compression methods, possibly layered (ie: RLE, VARINT, DEFLATE, ZSTD, etc.).
• ...

[dougbinks]

I think some of this is out of scope of a file format, and some of this is out of scope due to complexity. This isn't an attempt to create a universal file format for all voxel games, it's a file format for transferring voxel content between tools.

In detail:

• Mutability / editing: I think this is out-of-scope. This is for transferring content and not meant as the primary means of storage.
• Efficient indexing: I think non-finite is out-of-scope but we may need 64bit size/indexing, though volumes will likely be constrained to 32bit integer sizes.
• Non-discrete voxels As mentioned we should be able to support other parameters such as floating point, though these will likely not be well supported by all tools and OpenUSD with OpenVDB may be a better approach.
• Multiple layers This is either multiple scenes or models and thus supported that way. Whether we explicitly add layer support is TBD.
• Temporal Dimensions I think this is out-of-scope but could be handled by multiple scenes.
• Alternate Dimensions I think this is out-of-scope but could be handled by multiple scenes.
• Higher Dimensions I think this is out-of-scope.
• Holes & Sparsity (ie: infinite world generation). Infinite worlds are out-of-scope but holes and sparsity can be handled by multiple models under one node.
• Incompleteness (ie: phased world generation). I think this is out-of-scope. The file represents a moment in time.
• Arbitrary spatial depth/scale (ie: octrees). I think this is out-of-scope.
• Different data layouts over depth/location. I'm not sure what this means exactly.
• Compression methods, possibly layered (ie: RLE, VARINT, DEFLATE, ZSTD, etc.). I think we will likely require some form of simple compression to be specified in the format due to how large even simple volumes can get. TBD.

[Longor1996]

Woops, sorry! Copying old notes over without checking them was a bad idea. :'D

• Mutability: Agreed, doesn't matter for interchange.
• Indexing (mixed up with sparsity): Yes, indexing needs to be 64-bit (allow different modes?), even if it's just for "woops big coordinates".
• Non-discrete: Yeah, this is a question of "could be handled by OpenVDB", so not sure if necessary, but could be handled via a channel mechanism anyhow.
• Multiple layers: This should, at the very least, be allowed "per model" as per / via a channel mechanism.
• The dimension-y stuff was just there to showcase how many different 'axes'/dimensions volumes can have... so yes, that can be handled by multiple scenes / linked scenes; discard that.
• Holes/Sparsity: Yes, this can be handled by multiple models per node, but ought to have a standard as to how, in some way.
• Incompleteness: Especially if/because a file represents a moment in time, it should be possible to add/mark this.
• Arbitrary Spatial Depth/Scale: The scene description/graph allows for this, so... not really out-of-scope, just defined differently?
• Different data layouts over depth/location: This can-be/is handled by a channel mechanism; disregard.
• Compression Methods: Yes.

To summarize:

• Scene hierarchy/graph handles a lot of it.
• Alternate/linked models handle yet more.
• Channel mechanism / voxel properties (if allowed per model) covers the rest(?).

---

The only thing I can't agree with, is not having a way to define a temporal dimension... animations are a common thing to want to exchange after all! Forcing that over multiple files will cause quite a lot of copying, when only small bits of the volume(s) actually change.

EDIT: (animation) ...Apparently I can't read today.

---

EDIT 2: (indexing) ...nor write without mixing up more things. Sorry.

[Thibaulltt]

What's the goal of this new format ? Is it:

• having a new open format with good reference implementations (all for it, but shouldn't be the only thing) ?
• to have a really compact voxel representation on disk (i.e. having compression algorithms in the format as PNG/MPEG) ?
• to have a human-readable format for the metadata ?
• something else ?

[dougbinks]

As per the first sentence I'm hoping to create an open Voxel interchange/transfer format the goal is an open format for transferring models and scenes between voxel editors and engines.

It's not a goal to have a really compact voxel representation on disk, nor to have a human-readable format for the metadata.

[Longor1996]

nor to have a human-readable format for the metadata.

That seems backwards? Part of what makes glTF (which is mostly metadata) being JSON so useful, is that it's trivial to parse by humans and machines.

And if not JSON (or similar), what else? Is defining an entire tagged-binary-tree (or similar) format, just to store metadata with it, a good use of time/effort?

[dougbinks]

So it's not a goal to be human readable, nor is it a goal to be not human readable.

If we use glTF the metadata would be human readable (though just barely as indices are used rather than names so it's somewhat awkward to parse by human).

[AstrorEnales]

First of all, thank you for initiating this discussion and i'm all for it! I already agree with many of the points made.
Regarding animations, as this is probably the most important to me, i would advocate for:

1. Supporting frame-based animations
2. Supporting bone/armature definitions like glTF with animation tracks
3. Supporting multiple animations, not just one

Those are just some quick things immediately coming to my mind.

[dougbinks]

An extra note here on things I've read here/elsewhere:

1. If we were to extend something like glTF then with the right attributes a voxel scene could be made up of voxel models containing integer indices which reference a mesh model, thus supporting models similar to something like Minecraft's terrain.
2. Similarly a voxel could index another voxel model, and so support scenes from games like VoxIsle where every block is a .vox model.

[Longor1996]

Possibly relevant to (1): KhronosGroup/glTF#2496

(should this be a separate topic/comment?)

[dougbinks]

No, that's a good thing to add here if we end up using glTF.

[mgerhardy]

Just also dumping this here:

https://pro.arcgis.com/en/pro-app/latest/help/mapping/layer-properties/supported-voxel-formats.htm

The time components on voxel layers is also kind of interesting.

Also this documentation might give some general ideas: https://eisenwave.github.io/voxel-compression-docs/

And also the way to store the palette must be defined - colors could have names, materials and so on.

[dougbinks]

For materials if we use the glTF approach then this is an array of named materials, so we can either use array index or a separate palette of names per model, perhaps using different attribute names to clarify the difference.

[Longor1996]

Do note that, while the Eisenwave Voxel Compression page states that:

The goal of the project is to develop an efficient algorithm for compressing voxel models.

The input format it uses to evaluate said compression methods is... unfortunate, to say the least:

As an input format, an unsorted list of 3D integer coordinates and attribute data is used.

Pretty much the same concept that MagicaVoxel uses; effectively point clouds.

[scallyw4g]

Hi Doug, thanks for starting this conversation.

From your list of initial bullet points, the first two really stand out to me.

• As simple as possible - implementations should be able to be lightweight
• Open and extensible

I propose that these two requirements could form the basis of a successful format. We've seen open formats dominate in the past; glTF, OpenGL and Vulkan being good examples.

Another comment in the thread that jumped out me was:

I'll add that it'd be nice to make as many aspects of this format ignorable as possible, to simplify the process of writing an importer or exporter for an application that only cares about a subset of the features.

I think this is a useful idea, and makes for an attractive method of piece-wise writing reference implementations, and the specification. My preferred method of development looks something like the following:

1. Synthesize ideas for implementation
2. Write reference implementation
3. Formalize Specification based on learnings from reference implementation

I've used this method for writing compilers (with formal specifications) in the past, with good success.

---

In terms of the actual format discussion, there does exist a fourth option, which is to define our own custom format. My read on the options presented so far is as follows:

1. MagicaVoxel. Proprietary, several glaring technical limitations; worst of the three for me.
2. glTF. Format is relatively simple, extensible, and has a well supported ecosystem of tooling surrounding it. Best candidate.
3. USD / OpenVDB. These formats look to be (for my purposes at least) far too general/complicated.

Of the above options, (2) glTF seems like the best candidate to me. This comes with two large caveats; first, glTF appears to broadly operate under the assumption that it is a wire-transfer format, and will be gzip/deflate'd. It does not appear to have any native notion of compression, which we need. Secondly, after briefly reading the specification, I propose that we would not actually re-use much of the existing glTF spec, and instead we would end up stuffing our own specification format to fit inside glTF-shaped containers. While this is not a deal-breaker, it does sound to me like we may be introducing significant friction into our lives, for yet-unknown benefit. I could be wrong about this; I only read the spec for 30 minutes, but that's my initial read.

Briefly, it appears we may be able to reuse the following features to store arbitrary voxel data & scene descriptions.

1. Accessors & Buffers
2. Animation ..? (unclear to me if this could be used)
3. Asset (unclear, but possibly necessary)
4. Camera
5. Material
6. Scene/Node
7. Texture

I also read through the EXT_primitive_voxels proposed spec linked above and .. it looks pretty watery to me. It does appear to specify something along the lines of what we're discussing in this thread, but so far looks pretty half-baked, with some academic-looking frills that make me a bit nervous (the implicit volume stuff).

The real boon that I see glTF providing is that if we adopt that format, the barrier for applications to bake the voxel representation of the world into traditional triangulated 3D meshes, textures, etc is much lower. The resulting exports could be loaded into traditional 3D editors (Maya, Blender, etc). That said, then we're more talking about an export target for applications, and not necessarily a voxel data interchange format. That observation makes the value proposition of using something like glTF basically boils down to potentially sharing a parser with another export path, which isn't nothing, but isn't much either.

--

So.. that all said, I would propose adding a "novel voxel format", option to the list of candidates, with the following qualities:

Pros

• Maximum flexibility/control for our specific use case

Cons

• More decisions to be made
• No existing libraries for parsing (although we could just say it's JSON, which mostly solves this)

Thoughts?

I'm between jobs right now and could spend a day to write a simple .vox->.shiny_new_format utility at some point, if you're interested in code contributions.

[scallyw4g]

I would add that I actually strongly prefer binary formats to text-based formats, but I'm not opposed to starting with text-based and have a binary version if/when parser performance becomes an issue.

[dougbinks]

I think it's possible you might have slightly different goals than I do here, which hopefully we can integrate but it does mean there are trade offs which might make the solution non-optimal for each use case individually.

In my use case I'm likely to re-use a large part of the glTF spec (if we use that, or define something similar if we don't), and indeed hopefully benefit from the power of tools already designed around it. Animation support being one example - we could specify a fallback mesh (or KHR_implicit_shapes :) ) which could allow applications like Blender to import and animate a hierarchy. Admittedly we could also get this from our own format with a convertor tool.

I note that you stated voxel data interchange format. This is only part of the spec I'm interested in defining here, as per the requirements for my needs (and those of exchanging files with other editors like Vengi and engines like IOLITE or Voxile). We're likely not going to be able to implement the best voxel data interchange format since we want to remain fairly lightweight and simple to use, but hopefully we'll end up with something more optimal than a raw binary array or xyz 8bit coordinates.

[dougbinks]

I should add that I also don't like the EXT_primitive_voxels proposal, since it places voxel data in a mesh stream. They do so to leverage EXT_meshopt_compression which is a pretty sound general purpose compressor for floating point data, but we can likely do better for integer voxel data.

[scallyw4g]

In my use case I'm likely to re-use a large part of the glTF spec [ ... ] Animation support being one example

Cool. I don't have any particular interest in exporting animation data at the moment, nor do I have the technical knowledge to form an opinion on what I think are good features re. export/format options, so I might be asking a lot of questions around that.

which could allow applications like Blender to import and animate a hierarchy

I do agree this is the real power of extending something like glTF. It's unclear exactly how this would work in practice. Are we talking about simply animating object transforms (pos,rot,scale), skeletally rigging & animating 3D mesh data, or defining keyframe animations (effectively playback 3D video as voxels) ? Maybe some blend of the above?

voxel data interchange format. This is only part of the spec I'm interested in defining here

Agreed. I will say it could be easy to get lost in the weeds and blur the line between export format and voxel interchange format .. the former being more general, the latter being specifically about voxel data.

---

If we chose to extend glTF, do you have specific ideas for getting started?

[Longor1996]

This is most certainly pointless bikeshedding for now, but:

Since the format is mainly intended for the 'transfer' of voxel(scene)s, similar to glTF, would it make sense to call the format vxTF?