Support Visual Studio Ctrl+F7 (Compile Single File) Workflow with FastBuild .vcxproj generation

[missmah]

This is a feature request to add support for the Visual Studio Ctrl+F7 (Compile Single File) Workflow with FastBuild .vcxproj generation.

Currently FastBuild .vcxproj simply generates the following XML for a .cpp file:
<CustomBuild Include="File.cpp" />

I believe the solution to supporting CTRL+F7 to build an individual .cpp file would involve setting up a build step for each .cpp file, with an associated command to run (e.g. FBuild.exe $(ProjectName)-$(FileName)-$(Platform)-$(Configuration)).

I believe that this per-file build step should also not run during normal builds, because those are run through the NMake Build and Rebuild command lines, though I'm not fully positive of this.

@ffulin Is this something you have on your radar? Is it something we could implement and submit a pull request to you after we're done, with a little guidance on the best way to accomplish it?

Thanks!

[missmah]

It looks like in theory, something like this should work:

<CustomBuild Include="File.cpp"> 
      <Message>Compiling %(Identity) [%(FullPath)]</Message>  
      <Command>$(FASTBUILD_ROOT_DIR)\FBuild.exe $(ProjectName)-%(Identity)-$(Platform)-$(Configuration)</Command>  
      <Outputs>$(OutDir)%(Identity)</Outputs>       
</CustomBuild>

However, this seems to be ignored by Visual Studio in my preliminary tests. It's possible that VS by default makes a custom build command incompatible with an NMake project?

[missmah]

So, if you make a configuration which is a 'Utility' project, instead of a 'Makefile' project, then the above works.

You also need to make sure headers are flagged as C++ Headers in that case, so that the system doesn't attempt to build them:
<ClInclude Include="File.h" />

Finally, file paths also need to be relative, not absolute, or else the CustomBuild commands won't work.

I'm tempted to make an Iteration configuration, which works this way. However, there's still a remaining snag - I can't write the following code, because FastBuild can't get all the .cpp files in the project into a string for me:

ForEach( .cppFile in .ProjectCppFiles )
{
	// Compile Single Module (CTRL+F7 behavior in Visual Studio)
	ObjectList( '$ProjectName$-$cppFile$-OBJ-$Platform$-$Config$' )
	{
		.CompilerOptions			+ .ConfigurationCompilerOptions
		//							+ .ConfigurationPostPCHCompilerOptions // PCH Temporarily disabled due to bug
		.CompilerInputPath			= '$ProjectPath$\'
		.CompilerInputFiles			= '$cppFile$'
		.CompilerOutputPath         = '$OutputBase$\$ProjectName$\'
	}
}

One way to do this, is a new FastBuild function (C-like-pseudo-code):
BuildFileList( array of strings Dest, string Path, string IncludePattern, string ExcludePattern, bool bRecursive )

Such that the code could be:

BuildFileList( '$ProjectName$-CppFiles-Alias' )
{
    .Target = '.ProjectPath
    .IncludePattern = '*.cpp'
// .ExcludePattern = ; (optional)
    .RecurseDirectories = true
}

And then a another new function (I haven't come up with a good name yet), which would be able to build object lists for each file in the file list, such that it would be possible to invoke the compiler on any single object for very quick iteration time when working in a single file...

Ideally, BuildFileList would be able to write to a FastBuild array of strings, which could be used in the ForEach loop far above, instead of making an alias for the files as shown directly above; however, I don't think anything supporting those semantics (of a function call) really exists in FastBuild yet, and I'm guessing there's a strong reason for that, and as such, the alias that gets used by some other function is probably the way this would have to be implemented?

[missmah]

Another option is to invoke MSBuild for the CTRL+F7 case instead of augmenting FastBuild to handle compiling single files. Maybe we can spit out all the FastBuild parameters which we use for cl.exe and embed those into the project for use with a native MSBuild configuration instead? e.g. x64-MSBuild...

[missmah]

One more variation: if during Visual Studio Solution generation, we could call the code which is used for -showcmds for the platforms and configurations we cared about, we could embed that code into the .vcxproj as a custom build tool for each file (and specialize per configuration and per platform) (roughly this code).

    if ( FLog::ShowInfo() || ( FBuild::IsValid() && FBuild::Get().GetOptions().m_ShowCommandLines ) || isRemote )
    {
        output += useDedicatedPreprocessor ? GetDedicatedPreprocessor()->GetName().Get() : GetCompiler() ? GetCompiler()->GetName().Get() : "";
        output += ' ';
        output += fullArgs.GetRawArgs();
        output += '\n';
    }

The last step to make that work would be having a way to conveniently switch between the FastBuild Makefile versions and the MSBuild Utility project versions...

[missmah]

At long last, I got this working. It required some modifications to FastBuild, so it would be really nice to get these into a PR and get them into the main production build at some point, as I suspect other people may also find them valuable.

In the end, the solution was multi-fold:

1.) Have VSProjectGenerator.cpp mark headers as: <ClInclude Include="Filename.h" />

2.) Have VSProjectGenerator.cpp build a matrix of <Command Condition=*></Command> lines (specialized on $(Config) and $(Platform)).

- 2a.) Replace %1 and %2 in the CompilerOptions with %(FullPath) and $(OutDir)%(Identity), respectively.

- 2b.) Write the Compiler and updated CompilerOptions strings as the command (with some PATH boilerplate first, to ensure the compiler can be located).

3.) Add a new VCXProject option: .ProjectBuildType ; (optional)

4.) Create two .vcxprojects per project: $ProjectName$-proj and $ProjectName$-Iteration-proj
- 4a.) Make $ProjectName$-proj contain no files, and only exist for the NMake commands.
- 4b.) Make $ProjectName$-Iteration-proj contain all the files, and set it's .ProjectBuildType = 'Utility'
- 4c.) Make a solution containing these, and set only the $ProjectName$-proj projects as solution dependencies.

5.) Profit! Now, CTRL+SHIFT+B builds the NMake projects (All, or a selected one), and CTRL+F7 builds any .cpp file you have open inside the project (because they always load from the *-Iteration-proj.

This is handy because with Unity builds, sometimes changes in headers will force rebuild of whole Unity groups (even if you're careful about #includes). If you want always be able to have fast iteration time on one or a few .cpp files, while being able to touch headers and such, this makes it much easier to do so.

Another benefit is that this still works perfectly with the whole matrix of Configurations and Platforms you support. It will build a command line for the .cpp (or .c) file you're building based on the current Configuration and Platform.

Note: One minor current downside to this approach is that the output file it's generating is Filename.cpp.o, whereas FastBuild is generating Filename.o (as an example) - future work would be to make the output filenames match perfectly (find a way to strip the .cpp from the MSVC %(Identity) variable, so that any single-file builds you do would count towards your batch build, and not be discarded. Actually, thinking about this, some inline batch scripting inside the command can probably do it easily...

[dummyunit]

Just want to share my solution (or rather workaround) to Ctrl-F7 problem. Its big upside is that it doesn't require changes neither in FASTBuild nor in your .bff files, and the downside is that it doesn't work with unity builds.

First of all, FASTBuild already can(*) build individual files if you specify full path to resulting .obj file as target:
FBuild C:\myrepo\libfoo\x86-debug-whatever\file.obj
So my solution consists of three steps:

1. Add this powershell script to the root of your repo (lets call it build-item.ps1):

if ($Args.Count -lt 2)
{
	$ScriptName = $MyInvocation.MyCommand.Name
	Write "Usage: $ScriptName `$(TargetDir) `$(ItemPath)"
	Exit -1
}
$TargetDir = $Args[0]
$ItemPath = $Args[1]
$RelativeItemPath = $ItemPath.Replace($(Split-Path -Parent "$TargetDir"),"")
$Target = "$TargetDir$RelativeItemPath" -replace "\.(?:cpp|cxx|cc|c)$", ".obj"
& fbuild -vs -cache "$Target"

2. Add new external tool to VS (TOOLS/External Tools...) with options:

   • Command: powershell.exe
   • Arguments: -ExecutionPolicy RemoteSigned $(SolutionDir)\build-item.ps1 $(TargetDir) $(ItemPath)
   • Initial directory: $(SolutionDir)
   • Use Output window: true

3. Assign hotkey to this new external tool: Open TOOLS/Options..., open section Environment/Keyboard, find entry Tools.ExternalCommand with number corresponding to the place on which new external tool is listed.

(*) There is one caveat to that. If FASTBuild never attempted to build that .obj file since last .bff reparse it will complain that this build target is unknown.

[ffulin]

I have experimented with getting Ctrl+F7 working before, but I didn't have any luck. The approach I would be most likely to take next would be to create a VS plugin. My thought is that this would allow us to intercept the Ctrl+F7 and simply invoke FASTBuild with a special command line directly. I think some functionality would need to be added to FASTBuild to be able to "find" the correct target based on the combination of cpp file and normal project target. Something like this:

fbuild -buildsinglefile $(Config)-$(Platform)-$(Project) thing/stuff/file.cpp

The end result would be pretty close to what @dummyunit has, although potentially we could also potentially make it work even if you had never built the obj for that before.

I was actually thinking that the visualizer plugin would be a starting point for that (which I'm still working on integrating - I hit a snag because it seems it's simply not possible to build xaml without msbuild, and I don't want FASTBuild compilation to be dependency on msbuild).

[claimred]

I wonder if it is still possible to use PowerShell workaround when using Unity builds. Did anyone have any luck?

[aganea]

Ping! @ffulin Any plans for working on this? We'd like to move off MSBuild completely (to only keep FastBuild targets), and this issue prevents us for doing so. We already have a VS plugin that could trigger the fbuild cmd-line. What is missing is simply a flag to ask FASTBuild to compile a single file. That needs to be working with unity files as well (ie. if I ask to compile a .cpp swallowed by a unity that was never compiled before, it should work)
Thanks! :-)

[claimred]

We managed to use @dummyunit powershell build-item.ps1 script combined with dynamic unity autoisolation using .UnityInputIsolateListFile = 'unity_isolate_list' and another powershell script (using git):

$isolate_file = 'unity_isolate_list'
if (!(Test-Path $isolate_file))
{
  ni -Force $isolate_file | Out-Null
}
Clear-Content $isolate_file
git status --porcelain | foreach { $_.Substring(3) } | Set-Content $isolate_file

Still, it doesn't work when .obj file in question never existed but it doesn't really bother anyone that much. ('pull -> hit 'build all '-> make changes -> hit ctrl-f7 -> translation unit gets isolated' is the most common workflow I guess)

[ffulin]

I'd certainly like to support this, but it's not actively being worked on. (The accompanying plugin work I would do comes with a lot of other possibilities i'd like to explore to have a better integration as well)

To implement just the FASTBuild side alone is already complex though, esp. when considering the edge cases it needs to handle to be truly useful (like building for the first time and working out which targets a cpp file belongs to).

Most developers (myself included) use FASTBuild targets exclusively and use the "Isolate writable files" with Perforce and stick to "Solution Builds" to iterate. Similar to @claimred's comment, after the first build it has proved to be fast enough that most people don't seem to miss the Ctrl+F7.

I will consider your ping an upvote for this feature.

[aganea]

Thanks Franta! :)

[leo-polymorph]

Since VS 2022, NMake seems to now expose the Compile File Command Line:

I did not read the entire thread here, but it feels like it could simplify the way to deal with this initial feature request?

I am myself very interested in being able to compile a single file out of the unity pipeline. Mainly to locally verify that a cpp compile on itself, and is not lacking any include or forward declaration.

If I find some time to test that, I'll maybe give a try to mimic the current fastbuild ".ProjectBuildCommand" behavior and post my result here.

[ffulin]

@leo-polymorph Thanks for mentioning this here. I wasn't aware that had been added.

This solves half of the problem, but an important half. We'd no longer need to write a plugin to add this functionality. We'd instead be able to invoke FASTBuild with a unique command line directly.

The other half that still needs doing is converting those command line args (the config, platform and source file) into targets to build, which will involve some hoops to jump through to handle all cases, esp when a library in a config has never been built before.

In any case, this new feature this a lot easier to implement, so that's exciting.

(It looks like some additional changes to project file generation would be required - the files need to be "ClCompile" instead of "CustomBuild" in order for the option to not be grayed out)

[leo-polymorph]

Thanks for your answer @ffulin.

I had the first step working on my fastbuild fork repo: 1a360cf

As you pointed out, It's not enough for a full and easy-to-use ctrl+F7 integration, but in my case it's already enough to have a functional per-file compilation in my project.

Once I have this CompileFileCommand parameter, I can use it in this kind of way:

In the VSXProject:

.CompileFileCommand = 'set SELECTED_FILE=%1&FBuild.exe -ide -config "fbuild-single-file.bff" "single_file_libCore_Debug"'

Then I create a dedicated fbuild-single-file.bff:

#include "fbuild.bff"

#import SELECTED_FILE

ObjectList("single_file_libCore_Debug")
{
    Using(.obj_libCore_Debug)
    .PreBuildDependencies = {}
    .CompilerInputUnity = {}
    .CompilerInputFiles = { '$SELECTED_FILE$' }
}

So technically I'm dynamically generating a target for building one single file, the file name being forwarded as the environment variable SELECTED_FILE.

My main fbuild config file is fbuild.bff, and obj_libCore_Debug is the mutualized configuration. I'm just overriding CompilerInputFiles to only compile the single file, I'm clearing the CompilerInputUnity and I'm clearing PreBuildDependencies, because in my case I don't need dependencies to compile a single file.

I guess that for the full version of this feature you would envision a new command line argument to specify the single file to use, and FBuild would kind of filter out any input that is not this file? (In addition of behaving like -nounity).

I don't think I'll have the fbuild code base knowledge to go so far, especially because I can live with my work around. But let me know if my commit deserves a pull request, and/or if there is anything you feel I should address differently in it. I have some doubt about switching completely some CustomBuild to ClCompile for instance, and I saw some code related to linux in the VSXProject that I kind of ignored for now.

[ffulin]

@leo-polymorph Your change looks good to me - I think that is the first step we need. The only thing missing is updating the html docs for the new option.

For the CustomBuild to ClCompile change, I think we should try it and see if any issues arise during this release cycle. I suspect things will be fine.

I think the full/ideal version of the feature that would be built on top of this, has the following complexities:
a) the cpp file can be in multiple libraries or targets (you'd want to build the one that contains the currently "selected" file in VS)
b) there are multiple build configs and platforms (you'd want to build the current platform and build config)
c) if we've never build the library/platform/config flavor before, or nothing at all, we'd need to at the very least emit a sensible error or, time and complexity permitting, try to partially build the selected library such that it creates the nodes necessary to target the single file.

I think this part of the feature will be quite involved, but we don't need that to get your foundational changes in-place. The documentation for the new exposed property can have a disclaimer perhaps (indicating that it's basic/experimental and may be improved in the future)

[lucianm]

Since VS 2022, NMake seems to now expose the Compile File Command Line:

I did not read the entire thread here, but it feels like it could simplify the way to deal with this initial feature request?

I am myself very interested in being able to compile a single file out of the unity pipeline. Mainly to locally verify that a cpp compile on itself, and is not lacking any include or forward declaration.

If I find some time to test that, I'll maybe give a try to mimic the current fastbuild ".ProjectBuildCommand" behavior and post my result here.

Did you manage to get VS2022 actually execute whatever written there? If I edit my fastbuild-generated project just for testing purposes, I'm not getting any activated "Compile" (or Ctrl+F7) entry in VS2022....

Later Edit: "Compile"/Ctrl+F7 is only active for files "properly" referenced within the *.vcxproj file, so not as CustomBuild item like fbuild generates the VCXProject for integrating itself via the NMake mechanism, but as an ClCompile item, just like in native VC projects. This rises the question if it wouldn't be better to change the way how fbuild generates its VCXProject, maybe distinguishing by .ProjectAllowedFileExtensions like in the current implementation and a new .ProjectAllowedFileExtensionsForSingleBuild between files to be only referenced in the project, and those which should be included in proper ClCompile items allowing to build them individually with the provided command (which may accommodate a dynamic target like the one proposed by @leo-polymorph), or again, extended functionality in fbuild like @ffulin proposed with -buildsinglefile.

[lucianm]

Thanks for your answer @ffulin.

I had the first step working on my fastbuild fork repo: 1a360cf

As you pointed out, It's not enough for a full and easy-to-use ctrl+F7 integration, but in my case it's already enough to have a functional per-file compilation in my project.

Once I have this CompileFileCommand parameter, I can use it in this kind of way:

In the VSXProject:

.CompileFileCommand = 'set SELECTED_FILE=%1&FBuild.exe -ide -config "fbuild-single-file.bff" "single_file_libCore_Debug"'

Then I create a dedicated fbuild-single-file.bff:

#include "fbuild.bff"

#import SELECTED_FILE

ObjectList("single_file_libCore_Debug")
{
    Using(.obj_libCore_Debug)
    .PreBuildDependencies = {}
    .CompilerInputUnity = {}
    .CompilerInputFiles = { '$SELECTED_FILE$' }
}

So technically I'm dynamically generating a target for building one single file, the file name being forwarded as the environment variable SELECTED_FILE.

My main fbuild config file is fbuild.bff, and obj_libCore_Debug is the mutualized configuration. I'm just overriding CompilerInputFiles to only compile the single file, I'm clearing the CompilerInputUnity and I'm clearing PreBuildDependencies, because in my case I don't need dependencies to compile a single file.

I guess that for the full version of this feature you would envision a new command line argument to specify the single file to use, and FBuild would kind of filter out any input that is not this file? (In addition of behaving like -nounity).

I don't think I'll have the fbuild code base knowledge to go so far, especially because I can live with my work around. But let me know if my commit deserves a pull request, and/or if there is anything you feel I should address differently in it. I have some doubt about switching completely some CustomBuild to ClCompile for instance, and I saw some code related to linux in the VSXProject that I kind of ignored for now.

Cool, I was thinking about the same when stumbling upon this issue today, it seems pretty old already.

@ffulin , when would you adopt this, you seem to have found it well done? I think it would be already useful to be able to generate projects in which the path to $(SelectedFiles), (of which I would expect to contain just one file if right-cliking only on one) and let fbuild run the build command only on that one, I would need it badly together with undefining the FASTBUILD_CACHE_MODE environment variable in this command line, in order to check if compiler warnings from that specific file are gone, for example....

[ffulin]

I was hoping to add the "Compile File Command Line" support soon, but I didn't manage to have time to add it and test it yet. Exposing the option is trivial, but i think we might need some logic to keep the existing behavior for older VS versions just in-case.

I was kind of hoping to have a stab at the second half of this too - adding extra cmd line functionality to make the entire feature work without need for any special bff files etc. I have some ideas about how that could potentially work.

It might be best to just do the first half so at least there is something even if it's far from the desired end result.

> undefining the FASTBUILD_CACHE_MODE environment variable in order to check if compiler warnings from that specific file are gone

I think the real fix that is needed here is for cache items to capture warnings so they can be printed when retrieving from the cache.

Somewhat related, the cache is already disabled when building locally modified files individually outside of unity, so the fully functioning version as I imagine it would also avoid this problem.