SpvGenTwo

SpvGenTwo is a SPIR-V building and parsing library written in C++17, no other dependencies! No STL or other 3rd-Party library needed. The library comes with its own set of SPIR-V definitions generated from the machine readable grammar and therefore does not require any vulkan or spirv-headers includes. The generator can be found here: rustspvgen.

I built this library as a 'slim' backend for runtime material/shader-editors (like Proto) to avoid introducing enormous libraries like DXC (including LLVM and Frontend) or SPIRV-Tools to the codebase.

SpvGenTwo is still under development, many parts are still missing, but all the building blocks are there. SpvGenTwo is designed to be extensible and customizable, it is fairly easy to implement new SPIR-V instructions and extensions, use custom allocators and define own type inference rules. Note that it is still possible to generate invalid SPIR-V modules because not all inputs are checked yet. Use the SPIR-V validator spirv-val from the SDK and have the specification at hand while using this library.

I mainly focused on Shader capabilities, so the Kernel and OpenCL side is a bit under-developed. Any community contributions in that regard are very welcome!

Overview:

• Examples
  • Example Project
• Building
• Tools
  • Disassembler
  • Reflector
  • Linker
• Documentation
• Contributing
• Copyright and Licensing
• Roadmap
• Gallery
• Coverage
  • Core
  • GLSL.std.450
  • OpenCl.std.100

Examples

ConsoleLogger log;
HeapAllocator alloc; // custom user allocator

// create a new spir-v module
Module module(&alloc, &log);

// configure capabilities and extensions
module.addCapability(spv::Capability::Shader);
module.addCapability(spv::Capability::VulkanMemoryModelKHR);
module.addExtension(spv::Extension::SPV_KHR_vulkan_memory_model);
Instruction* ext = module.getExtensionInstructionImport(u8"GLSL.std.450");
module.setMemoryModel(spv::AddressingModel::Logical, spv::MemoryModel::VulkanKHR);

// global variables
Instruction* uniformVar = module.uniform<vector_t<float, 3>>(u8"u_Position");

// float add(float x, float y)
Function& funcAdd = module.addFunction<float, float, float>(u8"add", spv::FunctionControlMask::Const);
{
    BasicBlock& bb = *funcAdd; // get entry block to this function

    Instruction* x = funcAdd.getParameter(0);
    Instruction* y = funcAdd.getParameter(1);

    Instruction* z = bb.Add(x, y);
    bb.returnValue(z);
}

// void entryPoint();
{
    EntryPoint& entry = module.addEntryPoint(spv::ExecutionModel::Fragment, u8"main");
    entry.addExecutionMode(spv::ExecutionMode::OriginUpperLeft);
    BasicBlock& bb = *entry; // get entry block to this function

    Instruction* uniVec = bb->opLoad(uniformVar);
    Instruction* cross = bb.ext<GLSL>()->opCross(uniVec, uniVec); // use GLSL.std.450 extension
    Instruction* s = bb->opDot(cross, uniVec);
    entry->call(&funcAdd, s, s); // call add(s, s)
    entry->opReturn();
}

// custom spir-v binary serializer:
BinaryFileWriter writer("test.spv");
module.finalizeAndWrite(writer);

The resulting SPIR-V binary when disassembled using spirv-dis:

; SPIR-V
; Version: 1.0
; Generator: SpvGenTwo SPIR-V IR Tools(30); 0
; Bound: 20
; Schema: 0
               OpCapability Shader
               OpCapability VulkanMemoryModelKHR
               OpExtension "SPV_KHR_vulkan_memory_model"
          %1 = OpExtInstImport "GLSL.std.450"
               OpMemoryModel Logical VulkanKHR
               OpEntryPoint Fragment %main "main" %u_Position
               OpExecutionMode %main OriginUpperLeft
               OpName %u_Position "u_Position"
               OpName %add "add"
               OpName %main "main"
      %float = OpTypeFloat 32
    %v3float = OpTypeVector %float 3
%_ptr_Uniform_v3float = OpTypePointer Uniform %v3float
          %9 = OpTypeFunction %float %float %float
       %void = OpTypeVoid
          %3 = OpTypeFunction %void
 %u_Position = OpVariable %_ptr_Uniform_v3float Uniform
        %add = OpFunction %float Const %9
         %11 = OpFunctionParameter %float
         %12 = OpFunctionParameter %float
         %13 = OpLabel
         %14 = OpFAdd %float %11 %12
               OpReturnValue %14
               OpFunctionEnd
       %main = OpFunction %void None %3
         %15 = OpLabel
         %16 = OpLoad %v3float %u_Position None
         %17 = OpExtInst %v3float %1 Cross %16 %16
         %18 = OpDot %float %17 %16
         %19 = OpFunctionCall %float %add %18 %18
               OpReturn
               OpFunctionEnd

Example Project

Set CMake option SPVGENTWO_BUILD_EXAMPLES to TRUE to build included examples:

• FunctionCall Example
• ControlFlow Example
• FunctionCall Example
• Extensions Example
• Constants Example
• Types Example
• ExpressionGraph Example
• Linkage Example

Project Structure

SpvGenTwo is split into 5 folders:

• lib contains the foundation to generate SPIR-V code. SpvGenTwo makes excessive use of its allocator interface, no memory is allocated from the heap. SpvGenTwo comes with its on set of container classes: List, Vector, String and HashMap. Those are not built for performance, but they shouldn't be much worse than standard implementations (okay maybe my HashMap is not as fast as unordered_map, build times are quite nice though :). Everything within this folders is pure C++17, no other dependencies (given that SPVGENTWO_REPLACE_PLACEMENTNEW and SPVGENTWO_REPLACE_TRAITS are used).
• common contains some convenience implementations of abstract interfaces: HeapAllocator uses C malloc and free, BindaryFileWriter uses fopen, ConsoleLogger uses vprintf, ModulePrinter uses snprintf. It also has some additional classes like Callable (std::function replacement), Graph, ControlFlowGraph, Expression and ExprGraph, they follow the same design principles and might sooner or later be moved to lib if needed.
• example contains small, self-contained code snippets that each generate a SPIR-V module to show some of the fundamental mechanics and APIs of SpvGenTwo.
• dis is a spirv-dis-like tool to print assembly language text.
• refl is a SPIRV-Reflect-like tool to extract descriptor bindings and other relevant info from SPIR-V binary modules.
• link is a spirv-link-like tool to for merging symbols of modules into a new output module.

Building

Use the supplied CMakeLists.txt to generate project files for your build system. SpvGenTwo allows the user to use STL headers (<type_traits>, <new> etc) instead of my hand-made replacements (see stdreplament.h).

• SPVGENTWO_BUILD_EXAMPLES is set to FALSE by default. If TRUE, an executable with sources from the 'example' folder will be built.
  • Note that the SpvGenTwoExample executable project requires the Vulkan SDK to be installed as it calls spirv-val and spriv-dis.
• SPVGENTWO_BUILD_DISASSEMBLER is set to FALSE by default. If TRUE, an executable with sources from the 'dis' folder will be built.
• SPVGENTWO_BUILD_REFLECT is set to FALSE by default. If TRUE, an executable with sources from the 'refl' folder will be built.
• SPVGENTWO_BUILD_LINKER is set to FALSE by default. If TRUE, an executable with sources from the 'link' folder will be built.
• SPVGENTWO_REPLACE_PLACEMENTNEW is set to TRUE by default. If FALSE, placement-new will be included from <new> header.
• SPVGENTWO_REPLACE_TRAITS is set to TRUE by default. If FALSE, <type_traits> and <utility> header will be included under spvgentwo::stdrep namespace.
• SPVGENTWO_LOGGING is set to TRUE by default, calls to module.log() will have not effect if FALSE.

Note that I mainly develop on windows using clang and MSVC but I'll also try to support GCC/linux.

Tools

SpvGenTwo includes a couple of CLI tools to explore and test the libraries capabilities.

Disassembler

SpvGenTwoDisassembler source can be found at dis/source/dis.cpp, it is just a single source file demonstrating the basic parsing and IR walking capabilities of SpvGenTwo.

CLI: SpvGenTwoDisassembler [file] <option> <option> ...

Options

• -assignids re-assigns instruction result IDs starting from 1. Some SPIR-V compilers emit IDs in a very high range, making it hard to read and trace data flow in assembly language text, assignIDs helps with that.
• -serialize writes the parsed SPIR-V program to a serialized.spv file in the working directory (this is a debug feature).
• -noinstrnames don't replace result IDs with OpNames
• -noopnames don't replace operand IDs with OpNames
• -nopreamble don't print SPIR-V preamble
• -colors use ANSI color codes
• -tabs " " example: use 4 spaces instead of 2 tabs

Reflector

SpvGenTwoReflect source can be found at refl/source/refl.cpp

CLI: SpvGenTwoReflect [file] <option> <option> ...

Options

• -var name select variable by name (if OpVariable was annotated by OpName) for DescriptorType & Decoration printing (name has to be a UTF-8 string)
  • -var MyBuffer
• -deco decoration select decoration to query for in the module
  • -deco print all decorations in the module
  • -deco DescriptorSet to print only DescriptorSets
  • -deco DescriptorSet -deco Binding to print only DescriptorSets & Bindings
• -funcs list functions names in the module
• -vars list global variables in the module (StorageClass != Function)
• -types list types and constatns in the module
• -id Id print SPIR-V assembly text for the instruction with result Id
  • -id 24
• -colors use ANSI color codes

Linker

See SpvGenTwoLinker for detailed description. Source can be found at refl/source/link.cpp.

Documentation

Please read the documentation for more detailed information on how to use SpvGenTwo and some reasoning about my design choices.

Contributing

Please follow the "fork-and-pull" Git workflow:

• Fork the repo on GitHub
• Clone the project to your own machine
• Commit changes to your own branch
• Push your work back up to your fork
• Submit a Pull request so that we can review your changes

Make sure to merge the latest from "upstream" before making a pull request.

Copyright and Licensing

The SpvGenTwo open source project is licensed under MIT license. Any contribution you make to this original repository shall be licensed under same license. You are still free to distribute your contributions (in your own fork) under the license you desire.

Roadmap

A list of short and long term goals for this library:

• Implement more Instructions, at least 90% of Shader capabilities
• Improve validation / logging / error reporting
• Write unit tests
• Implement SPIRV-Tools like helper tools
• Implement some front-end or DSL like SPEAR

Gallery

SpvGenTwo is used in:

• Proto

• SHADERed

• TanTien-Engine (shadertrans)

Leave PR with your software if you want it to be added here :)

Coverage

SPIR-V IR generation progress, parsing is independent and auto generated. This table indicates whether the operation is already implemented in the Instruction class or can be generated by other facilities. This is not a complete list, if an entry is missing, assume it has not been implemented yet. You can file an Issue to request its implementation or just use Instruction::makeOp(...) directly.

Core

Instruction	Implemented
OpNop	✔
OpUndef	✔
OpSourceContinued	✔
OpSource	✔
OpSourceExtension	✔
OpName	✔
OpMemberName	✔
OpString	✔
OpLine	✔
OpExtension	✔
OpExtInstImport	✔
OpExtInst	✔
OpMemoryModel	✔
OpEntryPoint	✔
opExecutionMode	✔
OpCapability	✔
OpExecutionModeId	✔
OpTypeXXX	via Module::addType()
OpConstantXXX	via Module::addConstant()
OpSpecConstantXXX	via Module::addConstant()
OpSpecConstantOp	✔
OpFunction	✔
OpFunctionParameter	✔
OpFunctionEnd	✔
OpFunctionCall	✔
OpVariable	✔
OpImageTexelPointer	☐
OpLoad	✔
OpStore	✔
OpCopyMemory	☐
OpCopyMemorySized	☐
OpAccessChain	✔
OpInBoundsAccessChain	✔
OpPtrAccessChain	☐
OpArrayLength	☐
OpGenericPtrMemSemantics	☐
OpInBoundsPtrAccessChain	☐
OpDecorate	✔
OpMemberDecorate	✔
OpDecorationGroup	☐
OpGroupDecorate	☐
OpGroupMemberDecorate	☐
OpVectorExtractDynamic	✔
OpVectorInsertDynamic	✔
OpVectorShuffle	✔
OpCompositeConstruct	✔
OpCompositeExtract	✔
OpCompositeInsert	✔
OpCopyObject	✔
OpTranspose	✔
OpSampledImage	✔
OpImageSampleImplicitLod	✔
OpImageSampleExplicitLod	✔
OpImageSampleDrefImplicitLod	✔
OpImageSampleDrefExplicitLod	✔
OpImageSampleProjImplicitLod	✔
OpImageSampleProjExplicitLod	✔
OpImageSampleProjDrefImplicitLod	✔
OpImageSampleProjDrefExplicitLod	✔
OpImageFetch	via generic opImageSample
OpImageGather	via generic opImageSample
OpImageDrefGather	via generic opImageSample
OpImageRead	☐
OpImageWrite	☐
OpImage	☐
OpImageQueryFormat	☐
OpImageQueryOrder	☐
OpImageQuerySizeLod	☐
OpImageQuerySize	☐
OpImageQueryLod	☐
OpImageQueryLevels	☐
OpImageQuerySamples	☐
OpConvertFToU	✔
OpConvertFToS	✔
OpConvertSToF	✔
OpConvertUToF	✔
OpUConvert	✔
OpSConvert	✔
OpFConvert	✔
OpQuantizeToF16	✔
OpConvertPtrToU	✔
OpSatConvertSToU	☐
OpSatConvertUToS	☐
OpConvertUToPtr	☐
OpPtrCastToGeneric	☐
OpGenericCastToPtr	☐
OpGenericCastToPtrExplicit	☐
OpBitcast	✔
OpSNegate	✔
OpFNegate	✔
OpIAdd	✔
OpFAdd	✔
OpISub	✔
OpFSub	✔
OpIMul	✔
OpFMul	✔
OpUDiv	✔
OpSDiv	✔
OpFDiv	✔
OpUMod	✔
OpSRem	✔
OpSMod	✔
OpFRem	✔
OpFMod	✔
OpVectorTimesScalar	✔
OpMatrixTimesScalar	✔
OpVectorTimesMatrix	✔
OpMatrixTimesVector	✔
OpMatrixTimesMatrix	✔
OpOuterProduct	✔
OpDot	✔
OpIAddCarry	☐
OpISubBorrow	☐
OpUMulExtended	☐
OpSMulExtended	☐
OpAny	✔
OpAll	✔
OpIsNan	☐
OpIsInf	☐
OpIsFinite	☐
OpIsNormal	☐
OpSignBitSet	☐
OpLessOrGreater	☐
OpOrdered	☐
OpUnordered	☐
OpLogicalEqual	✔
OpLogicalNotEqual	✔
OpLogicalOr	✔
OpLogicalAnd	✔
OpLogicalNot	✔
OpSelect	✔
OpIEqual	✔
OpINotEqual	✔
OpUGreaterThan	✔
OpSGreaterThan	✔
OpUGreaterThanEqual	✔
OpSGreaterThanEqual	✔
OpULessThan	✔
OpSLessThan	✔
OpULessThanEqual	✔
OpSLessThanEqual	✔
OpFOrdEqual	✔
OpFUnordEqual	✔
OpFOrdNotEqual	✔
OpFUnordNotEqual	✔
OpFOrdLessThan	✔
OpFUnordLessThan	✔
OpFOrdGreaterThan	✔
OpFUnordGreaterThan	✔
OpFOrdLessThanEqual	✔
OpFUnordLessThanEqual	✔
OpFOrdGreaterThanEqual	✔
OpFUnordGreaterThanEqual	✔
OpShiftRightLogical	✔
OpShiftRightArithmetic	✔
OpShiftLeftLogical	✔
OpBitwiseOr	✔
OpBitwiseXor	✔
OpBitwiseAnd	✔
OpNot	✔
OpBitFieldInsert	☐
OpBitFieldSExtract	☐
OpBitFieldUExtract	☐
OpBitReverse	☐
OpBitCount	☐
OpDPdx	✔
OpDPdy	✔
OpFwidth	✔
OpDPdxFine	✔
OpDPdyFine	✔
OpFwidthFine	✔
OpDPdxCoarse	✔
OpDPdyCoarse	✔
OpFwidthCoarse	✔
OpEmitVertex	✔
OpEndPrimitive	✔
OpEmitStreamVertex	✔
OpEndStreamPrimitive	✔
OpControlBarrier	☐
OpMemoryBarrier	☐
OpAtomicLoad	☐
OpAtomicStore	☐
OpAtomicExchange	☐
OpAtomicCompareExchange	☐
OpAtomicCompareExchangeWeak	☐
OpAtomicIIncrement	☐
OpAtomicIDecrement	☐
OpAtomicIAdd	☐
OpAtomicISub	☐
OpAtomicSMin	☐
OpAtomicUMin	☐
OpAtomicSMax	☐
OpAtomicUMax	☐
OpAtomicAnd	☐
OpAtomicOr	☐
OpAtomicXor	☐
OpPhi	✔
OpLoopMerge	✔
OpSelectionMerge	✔
OpLabel	✔
OpBranch	✔
OpBranchConditional	✔
OpSwitch	☐
OpKill	✔
OpReturn	✔
OpReturnValue	✔
OpUnreachable	☐
OpNoLine	✔
OpAtomicFlagTestAndSet	☐
OpAtomicFlagClear	☐
OpImageSparseRead	☐
OpSizeOf	✔
OpTypePipeStorage	via Module::addType()
OpConstantPipeStorage	via Module::addConstant()
OpCreatePipeFromPipeStorage	☐
OpGetKernelLocalSizeForSubgroupCount	☐
OpGetKernelMaxNumSubgroups	☐
OpTypeNamedBarrier	via Module::addType()
OpNamedBarrierInitialize	☐
OpModuleProcessed	☐
OpExecutionModeId	☐
OpDecorateId	✔
...	...

GLSL.std.450

Instruction	Implemented
Round	✔
RoundEven	✔
Trunc	✔
FAbs	✔
SAbs	✔
FSign	✔
SSign	✔
Floor	✔
Ceil	✔
Fract	✔
Radians	✔
Degrees	✔
Sin	✔
Cos	✔
Tan	✔
Asin	✔
Acos	✔
Atan	✔
Sinh	✔
Cosh	✔
Tanh	✔
Asinh	✔
Acosh	✔
Atanh	✔
Atan2	✔
Pow	✔
Exp	✔
Log	✔
Exp2	✔
Log2	✔
Sqrt	✔
InverseSqrt	✔
Determinant	✔
MatrixInverse	✔
Modf	☐
ModfStruct	☐
FMin	✔
UMin	✔
SMin	✔
FMax	✔
UMax	✔
SMax	✔
FClamp	✔
UClamp	✔
SClamp	✔
FMix	✔
IMix	☐
Step	✔
SmoothStep	✔
Fma	✔
Frexp	☐
FrexpStruct	☐
Ldexp	☐
PackSnorm4x8	☐
PackUnorm4x8	☐
PackSnorm2x16	☐
PackUnorm2x16	☐
PackHalf2x16	☐
PackDouble2x32	☐
UnpackSnorm2x16	☐
UnpackUnorm2x16	☐
UnpackHalf2x16	☐
UnpackSnorm4x8	☐
UnpackUnorm4x8	☐
UnpackDouble2x32	☐
Length	✔
Distance	✔
Cross	✔
Normalize	✔
FaceForward	✔
Reflect	✔
Refract	✔
FindILsb	☐
FindSMsb	☐
FindUMsb	☐
InterpolateAtCentroid	☐
InterpolateAtSample	☐
InterpolateAtOffset	☐
NMin	☐
NMax	☐
NClamp	☐

OpenCl.std.100

Not implemented yet