Strange #include/file system behaviour

[simontaylor81]

I'm getting some strange behaviour with #includeing files from the file system using the dxc commandline app, which are somwhat hard to describe.

First, the repro:

First, download this file, rename .zip (github doesn't like zip files apparently), and extract somewhere. OddIncludes.txt
The repro relies on using absolute paths, so I can't just give a command line to run directly, but assuming the zip was extracted to c:\temp, run the following command (assumes dxc.exe is on the path):

dxc /T ps_6_0 c:\temp\OddIncludes\main.hlsl /I c:\temp

This should generate an error:

c:\temp\OddIncludes\main.hlsl:1:10: fatal error: 'include.hlsl' file not found
#include "include.hlsl"
         ^

This is erroneous -- include.hlsl is in the same directory as main.hlsl, so this should work fine. However, you can "fix" this error by changing the case of the directories. So this command works (note capital T):

dxc /T ps_6_0 c:\temp\OddIncludes\main.hlsl /I c:\Temp

Ok, so now the explanation. Apologies if this is very confusing! The first thing to note is that the (in this case superfluous) additional include directory specification is critical to the repro. Additional include directories are process before any source files (this matters because the file system has lots of caching). When it tries to get the file info for this path, it ends up in DxcArgsFileSystem::CreateFileW. That code does a substring search of the path against the input filename. If it matches (in this case it does), it assumes it is the parent of the source and returns the special handle SourceParentDirHandle. This is added to the cache with the path c:\temp.

Later on, we come to look up the directory for including the file. This is also a parent of the source file, and so also matches the substring match, and so also returns SourceParentDirHandle. This is then a cache hit, so instead of returning the directory asked for, it returns c:\temp again. Include.hlsl file is not found there, giving the error.

Now, the reason changing the case "fixes" things is that all the substring comparisons in dxcompilerobj.cpp are case sensitive! So by changing the case, the substring comparisons fail, and it falls through to just looking up the file in the file system, and things work correctly.

Hopefully that made some sense! Basically, I think the logic surrounding SourceParentDirHandle is broken. I do not know the best way to fix it, though, I'm afraid.

[marcelolr]

Thanks for reporting this, this is an awesome bug. There are a bunch of issues to consider, including providing distinct handle values, absolute-vs-relative references (plus, relative-to-what considerations), case sensitivity, and how much complexity / sophistication we push onto include handlers. I'm working on a fix right now, hopefully I'll have a fix along with some clarifications on design and expectations in the near future.