Include mechanism inherently tied to file system

[simontaylor81]

The new include file handling mechanism (i.e. IDxcIncludeHandler) seems to have significantly less expressive power than the old ID3DInclude. Specifically, it receives already-resolved absolute file system paths, not the raw paths specified in the source file.

Our shaders use virtual paths that operate using our internal virtual file system, and do not necessarily correspond to real file system paths. To use the new interface we will have to somehow invent dummy include search paths and reverse-map them to virtual paths. Previously we just got the raw path and passed it to the virtual file system directly.

Another common requirement is mapping "special" include names to files that only exist in-memory (e.g. they have been generated by some internal process). I believe Unreal does this currently, for example. This could be hacked in using the current mechanism by stripping the path information and just looking at the filename, but it seems ugly and error prone to have to do it that way.

[marcelolr]

@simontaylor81, I think I understand your concerns. You make some valid points and I want to address them here.

First, and just to get it out of the way, the include mechanism is tied to a file system abstraction, not a specific implementation. That's the point of the handler callback - but you already know this, as you're mentioning various possible implementations on how to have files available for inclusion based on other criteria, like compiling within a specific engine.

ID3DInclude's flexibility is problematic for downstream consumers. Tools that want to find include files can't rely on completely arbitrary behavior in general. Things like building a system to invalidate targets based on changed headers, or edit-and-continue, or implementing something like #pragma once become more difficult if we don't have all parts agreeing on a canonical name for resolved headers after compilation, regardless of how we got there.

Like you touch on, the file system abstraction is virtual, so presumably the same handler implementation could be made available to get consistent results in various scenarios (or saving some extra information during compilation to record contents associated with file names). The new interface is meant to be stateless; the prior one was a largely opaque state machine.

While you're now responsible for developing a consistent abstraction, we believe that if you agree up-front on where your virtual files will be laid out on your virtual system and how you'll refer to them, the implementation should be quite straightforward.

I agree this is different from the work that was required before, but there are more things enabled with this scheme and we think it's the right longer-term approach. For what it's worth, I believe it was very straightforward to get Unreal up and running w.r.t. includes.

[simontaylor81]

I managed to come up with a fairly decent compromise/workaround I think. I add as an argument to the compiler /Ivfs:/, which adds the (nonsense) include path "vfs:/". This is "path-like" enough to be accepted by the compiler, but bogus enough to not accidentally hit actual files. I can then look for that prefix in the include handler, and if it matches, pass the request to the virtual file system.

The only slight wrinkle is that relative includes will still come out as real file paths, but I hacked around that by just stripping the known VFS root path and treating it as a VFS path. This is a slightly odd case, because we pass a valid physical path to the compiler even though the data actually comes from the VFS because we want to have real filenames in the error messages (but still want to go through the VFS, for various reasons).

There are still a few things that I think are too tied to the physical file system rather than a file system-like abstraction -- for example, path matching is different based on the platform that the code is running on. However, I think we can make it work, even if it won't be the most elegant thing in the world :). Thanks for you help!

[marcelolr]

Closing this issue thread, keeping #161 as the main reference one (Wiki points to that one as well).