Repository navigation
Home
Ray tracing (RT) the the rendering technique used by modern games to provide a more realistic look. RT has a lot of advantages over the traditional raster rendering most game use to this day. RT makes the workflow much easier, no need to bake lighting which takes tons of time, allows for real time editing of lights and soft shadows essentially for free. While RT sounds like the better option it comes with some drawbacks, it is very performance heavy, each pixel on the screen needs to send out rays and calculate all the bounces until it reaches a bounce limit, or flies into a light source or the atmosphere. This takes lots of time to calculate, thankfully the hardware we have nowadays is up to the task, most graphics cards have dedicated hardware to handle these ray calculations.
Understanding the inner workings of a ray tracer is valuable as it will eventually be the preferred rendering technique for games, this is proven by some games that already demand a GPU that has dedicated hardware support for RT or it will refuse to even start.
System requirements for Indiana Jones and the Great Circle - Notice the "GPU Hardware Ray" under Additional Notes
I saw a great series by Sebastian Lague about RT and decided to base my project on his. I watched both his videos to get a general understanding of how I can create something similar. His project used glsl, but I was determined to use Compute shaders and hlsl so I looked at another video of his that implemented a very simple ray tracer in hlsl and started from there.
I initially started with an HDRP project, which immediately caused problems as I wanted to use the [ImageEffectAllowedInSceneView] attribute to render the raytracer's output to the scene camera for easy editing, but that is not possible in HDRP, after reverting to the Built-In renderer it was time to get started. My first milestone was to render spheres to the texture and then blit it to the camera's output. I started by defining a RayTracedObject this is the base script that describes an object that will be sent to the GPU for rendering. Next was creating a manager script that handles the buffers, textures and dispatching the compute shader. This was relatively simple, but the scene camera view caused a lot of headaches in the process as the script had to be marked with the [ExecuteAlways] attribute, this meant that the script was executed during edit mode too. Once this framework was in place i could move onto rendering spheres.
The first major challenge was creating the rays that will be sent out from the camera. For this I read the dimensions of the input texture, which was the screen size, this game me a float2 with the screen space coordinates of the given pixel. With the coordinates I could transform them into clip space coordinates and then finally with the camera's inverse projection matrix into a direction for the ray to travel. Now all I needed to do was multiply the direction by the camera's local to world matrix and I had a direction vector. With this I could create a ray for each pixel with the camera's position as the starting point, the previously calculated ray direction which goes through the cameras near plane and diverges with distance.
Ray ConstructPixelRay(uint3 id, uint index)
{
uint pixelID = id.x * width + id.y;
float random = RandomUInt(pixelID + _IterationCount + index);
Ray ray;
ray.origin = _CameraPosition;
float2 uv = (id.xy + float2(random.xx)) / float2(width, height) * 2.0 - 1.0;
float4 clipUV = float4(uv, 0, 1);
float4 rayView = mul(_CameraInverseProjection, clipUV);
rayView /= rayView.w;
float3 rayDirection = normalize(mul(_CameraToWorld, float4(rayView.xyz, 0)).xyz);
ray.direction = rayDirection;
return ray;
}Now that I had rays to shoot into the world, I started on sphere rendering by consulting an article about ray sphere intersections, and implementing it in hlsl. The math for this was relatively simple, using Pythagorean's theorem the intersection can be calculated, thus i could render spheres to the screen.
HitInfo SphereRayIntersect(Ray ray, Sphere sphere)
{
HitInfo hitInfo = (HitInfo)0;
float3 offsetRayOrigin = ray.origin - sphere.position;
float a = dot(ray.direction, ray.direction);
float b = 2 * dot(offsetRayOrigin, ray.direction);
float c = dot(offsetRayOrigin, offsetRayOrigin) - sphere.radius * sphere.radius;
float d = b * b - 4 * a * c;
if (d >= 0)
{
float distance = (-b - sqrt(d)) / (2 * a);
if (distance >= 0)
{
hitInfo.isHit = true;
hitInfo.distance = distance;
hitInfo.position = ray.origin + ray.direction * distance;
hitInfo.normal = normalize(hitInfo.position - sphere.position);
hitInfo.material = sphere.material;
}
}
return hitInfo;
}Intersection logic from article on Scratchapixel. (Scratchapixel, n.d.)
Next I wanted to have some basic shading, no path tracing just yet. This was relatively simple as I have done something similar in glsl before. If i had a sphere intersection, I got the point of intersection and sent a ray towards the light source's position, if there were no spheres intersected along the way the sphere was fully in light. If there was an intersection I dimmed the color of the circle by multiplying the color by an arbitrary value.
float3 Trace(Ray ray)
{
//Find the closest intersection with a sphere
HitInfo closestIntersection = FindClosestSphereIntersection(ray);
//Get the color of the sphere and store it for later use
float4 finalColor = closestIntersection.material.color;
//Create a ray from the closest intersection to the light source to get simple shadows
Ray hitPointRay;
hitPointRay.origin = closestIntersection.position + closestIntersection.normal * 0.001;
hitPointRay.direction = normalize(_LightPosition - hitPointRay.origin);
//Check if the constructed ray intersects anything on its way to the light source, if it is dim the color
if (FindClosestSphereIntersection(hitPointRay).isHit)
{
finalColor *= 0.2;
}
return finalColor;
}Next I wanted to implement multiple bounces to make the lighting look better, as well as making the scene camera work with the shader. Implementing multiple bounces was also a bit tricky, the biggest problem is that when we hit something we want the ray to bounce and keep going while accumulating the color of every object it hits. This is difficult on the GPU as we want the ray to bounce in random directions not just at the same angle as it's incoming angle, this would make all object be like a mirror and that is less then ideal. For this a hashing algorithm is required, a hashing algorithm takes in a number and jumbles it around to produce a semi-random number, we can never get a truly random number but as long as values that are close to each other end up far from each other after hashing this will not be a problem. Now all I needed to do was to let the ray bounce after an intersection and count up the colors it met along the way, then draw the color to the texture after it ran out of bounces or it flew off into the atmosphere
// PCG (permuted congruential generator). Thanks to:
// www.pcg-random.org and www.shadertoy.com/view/XlGcRh
uint RandomUInt()
{
state = state * 747796405u + 2891336453u;
uint word = ((state >> ((state >> 28u) + 4u)) ^ state) * 277803737u;
return ((word >> 22u) ^ word);
}
float RandomFloat01()
{
return RandomUInt() / 4294967295.0;
}Now to get the scene camera to render using the raytracer. This was in theory incredibly simple but ended up being the most annoying feature. Using the [ImageEffectAllowedInSceneView] attribute on the manager script's class I could hook into unity's OnRenderImage() function call to render with the raytracer when needed and blit the texure to the camera's output. The only major problem with this is managing the lifetime of objects like the ray traced objects that live in the scene, and that the Camera.current or Camera.main sometimes contain null references for seemingly no reason, thus getting errors for missing references was a common occurance.
The next big milestone I set was to render meshes. This was the most important to implement as with this I can render any 3D object I import. To start I read another Sketchapixel article about ray-triangle intersection math and implemented it in the shader.
HitInfo TriangleRayIntersect(Ray ray, Triangle tri, float4x4 localToWorld)
{
HitInfo hitinfo = (HitInfo) 0;
hitinfo.isHit = false;
// Compute the plane's normal
float3 v0v1 = tri.b - tri.a;
float3 v0v2 = tri.c - tri.a;
// No need to normalize
float3 N = cross(v0v1, v0v2); // N
// Step 1: Finding P
// Check if the ray and plane are parallel
float NdotRayDirection = dot(N, ray.direction);
if (abs(NdotRayDirection) < 1e-6) // Almost 0
return hitinfo; // They are parallel, so they don't intersect!
// Compute d parameter using equation 2
float d = dot(-N, tri.a);
// Compute t (equation 3)
float t = -(dot(N, ray.origin) + d) / NdotRayDirection;
// Check if the triangle is behind the ray
if (t < 0)
return hitinfo; // The triangle is behind
// Compute the intersection point using equation 1
float3 P = ray.origin + t * ray.direction;
// Step 2: Inside-Outside Test
float3 Ne; // Vector perpendicular to triangle's plane
// Test sidedness of P w.r.t. edge v0v1
float3 v0p = P - tri.a;
Ne = cross(v0v1, v0p);
if (dot(N, Ne) < 0)
return hitinfo; // P is on the right side
// Test sidedness of P w.r.t. edge v2v1
float3 v2v1 = tri.c - tri.b;
float3 v1p = P - tri.b;
Ne = cross(v2v1, v1p);
if (dot(N, Ne) < 0)
return hitinfo; // P is on the right side
// Test sidedness of P w.r.t. edge v2v0
float3 v2v0 = tri.a - tri.c;
float3 v2p = P - tri.c;
Ne = cross(v2v0, v2p);
if (dot(N, Ne) < 0)
return hitinfo; // P is on the right side
hitinfo.isHit = true;
hitinfo.position = P;
hitinfo.distance = t;
hitinfo.normal = tri.Anormal;
return hitinfo; // The ray hits the triangle
}Sketchapixel code for ray-triange intersection rewritten to use custom HitInfo struct and its values (Sketchapixel, n.d.)
There were a few problems with this approach though, each ray had to calculate its intersection with each triangle in the buffer, even if they are far from intersecting, thus tanking performance when using meshes with lots of triangles. I learned about Bounding Box checks and Bounding Volume Hierarchies to expedite this process, and for now just decided to implement the Bounding Box check. Another performance boost was from changing the intersection calculation for something faster, the Sketchapixel one is very transparent, and a lot easier to understand than some other intersection code but I decided to change to the one Sebastian Lague was using which was a lot faster.
HitInfo TriangleRayIntersect(Ray ray, Triangle tri, float4x4 localToWorld)
{
tri.a = mul(localToWorld, float4(tri.a.xyz, 1)).xyz;
tri.b = mul(localToWorld, float4(tri.b.xyz, 1)).xyz;
tri.c = mul(localToWorld, float4(tri.c.xyz, 1)).xyz;
float3 edgeAB = tri.b - tri.a;
float3 edgeAC = tri.c - tri.a;
float3 normalVector = cross(edgeAB, edgeAC);
float3 ao = ray.origin - tri.a;
float3 dao = cross(ao, ray.direction);
float determinant = -dot(ray.direction, normalVector);
float invDet = 1 / determinant;
//Calculate dst to triangle & barycentric coordinates of intersection point
float dst = dot(ao, normalVector) * invDet;
float u = dot(edgeAC, dao) * invDet;
float v = -dot(edgeAB, dao) * invDet;
float w = 1 - u - v;
HitInfo hitInfo;
hitInfo.isHit = determinant >= 1E-6 && dst >= 0 && u >= 0 && v >= 0 && w >= 0;
hitInfo.position = ray.origin + ray.direction * dst;
hitInfo.normal = normalize(tri.Anormal * w + tri.Bnormal * u + tri.Cnormal * v);
hitInfo.distance = dst;
return hitInfo;
}The last big performance hit was from having to perform 3 matrix multiplications for each intersection for each ray, this is incredibly expensive, so for now I made the CPU do the calculations once when the object was registered, and it's mesh data translated to the sequential format needed for the buffers.
With all that this was the current look of the renderer:
Now I needed to optimize the rendering as a simple unity primitive such as the capsule was tanking the fps to below 2 digits. The biggest optimization was to not do matrix multiplications on individual vertices, but to multiply the ray position and direction by the mesh's world-to-local matrices to transform and translate the ray to the mesh's local coordinate space, thus skipping all the per vertex multiplications. This was again in theory simple but ended up taking days and days to figure out why it wasn't working correctly.
Some images from the countless hours of headaches
In the end the fix was to not bundle the matrix within the mesh data struct but to create a separate buffer for the matrices and use an index to access the matrices. I am not sure why this was the fix, I had enough vram and free buffers yet this simple fix worked, and boosted performance by a lot. With this I could easily calculate a Bounding Box for each mesh and check for intersections with these first. Another quick and easy performance improvement was to split each mesh into several simpler meshes, each with a limited amount of triangles. This made sure that we did more Bounding Box intersection calculations rather then the more expensive triangle calculations.
Now onto the testing, I wanted to test my renderer against unity's Built-In renderer, and HDRP renderer with some settings and Light Probes. For this I set up scenes with differing objects and settings.
- The first scene was the sphere rendering, I had 5 spheres with different sizes and colors, 2 of which were set to be fully reflective. The scene also has a plane that the spheres rest on
- The second scene was the most demanding, it contains a box with differing colors, a mirror at the back, a knight chess piece model in the center and one light source. The scene has no environmental lighting, only the light in the box.
- The third scene is just a queen chess piece with a mirror behind it at an angle.
With these scenes I can test simple sphere rendering; which is a lot more performant in my renderer then meshes, I can test the visuals and performance of a locally lit box with reflections, and a simple reflection test with the 3rd scene.
With the scenes set up I turned my attention to writing a benchmark script. To make sure no outside noise can enter the testing results I was going to log the results in a built version of the renderer, so the Unity Editor doesn't affect the performance. To run the test I parse the command line arguments, and queue up the scenes to be tested, then each scene is loaded one after the other, left to warm up for a few seconds, then ran for a few seconds while saving the frame time each frame. In the end the results are written to a .txt file with some information like graphics driver, scene name, duration, warmup time, etc. Using this system I conducted the tests.
The first test I ran was comparing my ray tracer to Unity's Built-In renderer and then to Unity's High Definition Render Pipeline. I ran the benchmark on each 3 times and averaged the results. I also ran the tests on different graphics apis namely, DirectX11, DirectX12, and Vulkan. Here are the results:
| RT | Outdoor | Box | Balls |
|---|---|---|---|
| DX11 | 61,13fps | 6,05fps | 64,54fps |
| DX12 | 61,55fps | 4,68fps | 64,58fps |
| Vulkan | 79,73fps | 7,56fps | 70,93fps |
| HDRP | Outdoor | Box | Balls |
|---|---|---|---|
| DX11 | 3518,33fps | 3938,07fps | 3454,97fps |
| DX12 | 3559,48fps | 4414,98fps | 3348,05fps |
| Vulkan | 2303,89fps | 2883,55fps | 2285,72fps |
| Built-In | Outdoor | Box | Balls |
|---|---|---|---|
| DX11 | 1378,64fps | 577,15fps | 1592,61fps |
| DX12 | 1059,56fps | 322,19fps | 1254,09fps |
| Vulkan | 1078,96fps | 352,69fps | 1338,56fps |
The second test was comparing my ray tracer under different workloads, for this I create simple scenes with different models that have increasing vertex and triangle counts. Here is the table that shows the info about the models used:
| Verts | Tris | |
|---|---|---|
| Cube | 24 | 12 |
| Queen | 1160 | 582 |
| Dragon | 64700 | 23490 |
Using these scenes I ran the test 5 times, and used the 3 graphics apis mentioned before, then the results are averaged and imported into excel. Here are the results:
| Cube | Queen | Dragon | |
|---|---|---|---|
| DX11 | 3147,16fps | 668,15fps | 15,26fps |
| DX12 | 2591,65fps | 665,79fps | 15,40fps |
| Vulkan | 1960,34fps | 773,92fps | 21,30fps |
- The first test clearly shows that there is a lot more performance optimizations left to do for my renderer to be up to par with Unity's most expensive rendering pipeline only getting about 4-1% of the performance which is terrible, but the redeeming factor is that the output of my renderer is more visually pleasant, more realistic and easier to work with.
- The second test also clearly shows that there is a big drop off in performance with increasing the triangle count, but it was not as bad as I expected it to be. Looking at the fps and the triangle count between the cube and the queen, the triangle count going from the cube to the queen is 48.5x, but the fps is only down to 20% of the cube's fps, when looking at the DirectX11 numbers, I will get back to this later. Then going from the queen to the dragon is another 40.3x increase in triangles but the fps drops to just 2% of the queens fps, this is because we are adding another 40.3x triangles on top of the previous 48.5x, thus increasing the amount of Bounding Box checks we need to do for the mesh, even if we are not checking the individual triangles, this tanks performance.
- Now for fun let's compare the rendering apis. There is no clear winner across the board. In some cases DiretX11 was the clear winner, with substantially more fps compared to the others. From the data something surprising is that Vulkan consistently performs better under heavier loads then DirectX11 or DirectX12. In the first test Vulkan was the best performer when running my shader, and in the second test, the higher the triangle count got, the better it performed against the other two rendering apis. All the results can be found can be found in the form of a .xlsx file in the root of the repository!
- Sebastian Lague (Apr 1, 2023) Coding Adventure: Ray Tracing - Video https://www.youtube.com/watch?v=Qz0KTGYJtUk
- Sebastian Lague (Jun 12, 2024) Coding Adventure: More Ray Tracing! - Video https://www.youtube.com/watch?v=C1H4zIiCOaI
- Sebastian Lague (Mar 14, 2019) Coding Adventure: Compute Shaders - Video https://www.youtube.com/watch?v=9RHGLZLUuwc
- Branch Education (Aug 17, 2024) How does Ray Tracing Work in Video Games and Movies? - Video https://www.youtube.com/watch?v=iOlehM5kNSk
- Scratchapixel (n.d.) A Minimal Ray-Tracer - Article https://www.scratchapixel.com/lessons/3d-basic-rendering/minimal-ray-tracer-rendering-simple-shapes/ray-sphere-intersection.html
- Scratchapixel (n.d.) Generating Camera Rays with Ray-Tracing - Article https://www.scratchapixel.com/lessons/3d-basic-rendering/ray-tracing-generating-camera-rays/generating-camera-rays.html
- Scratchapixel (n.d.) Ray-Tracing: Rendering a Triangle - Article https://www.scratchapixel.com/lessons/3d-basic-rendering/ray-tracing-rendering-a-triangle/ray-triangle-intersection-geometric-solution.html