For incremental builds, should -Outputs allow empty lists?

[daviwil]

Hey Roman, I'm trying to use incremental builds for the PowerShell Editor Services build script and ran across this error:

Build FAILED. 3 tasks, 1 errors, 0 warnings 00:00:00.9779218
C:\Users\daviwil\Documents\WindowsPowerShell\Modules\InvokeBuild\3.2.1\Invoke-Build.ps1 : Outputs must not be empty.
At line:1 char:1
+ Invoke-Build BuildHost
+ ~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (Outputs must not be empty.:String) [Invoke-Build.ps1], RuntimeExcepti
   on
    + FullyQualifiedErrorId : Outputs must not be empty.,Invoke-Build.ps1

Since I'm using Get-ChildItem in my Outputs list to get the list of potential outputs, that list will be empty if those output files haven't yet been created. Is it possible that this error could be removed, or is there some other way around it?

Thanks!

[nightroman]

The way around is to return paths of the expected items.

Let me think if this requirement/error can be removed for simple incremental tasks. Presumably, it cannot for partial incremental.

[daviwil]

One workaround I've been trying to come up with is using Join-Path to generate the file paths which may not exist yet (before the first full restore or build of my .NET projects). Here's an example:

task Restore `
    -Inputs { Get-ChildItem -Recurse *.csproj } `
    -Outputs { Join-Path (Get-ChildItem .\src, .\test -Directory | Select-Object -ExpandProperty FullName) -ChildPath "obj\project.assets.json" } `
    -Before Build, BuildHost, Test {

    exec { & $script:dotnetExe restore }
}

Strangely this doesn't work for some reason, though, if I take the first item of the array returned by the ScriptBlock given to Outputs it does work:

task Restore `
    -Inputs { Get-ChildItem -Recurse *.csproj } `
    -Outputs { (Join-Path (Get-ChildItem .\src, .\test -Directory | Select-Object -ExpandProperty FullName) -ChildPath "obj\project.assets.json")[0] } `
    -Before Build, BuildHost, Test {

    exec { & $script:dotnetExe restore }
}

[nightroman]

@daviwil, can you describe your original outputs that did not work as you
wanted? Is it {Get-ChildItem ...}? I afraid, this way is not supported.

IB expects required paths for output, including missing paths, i.e. not
items excluding missing. Can you see the difference?

Example of the correct usage:

task X -Inputs ... -Outputs {'X.assets.json', 'Y.assets.json'} ...

Example of the tempting but not supported/correct usage:

task X -Inputs ... -Outputs {Get-ChildItem *.assets.json} ...

Suppose we support the second case. Then we have a problem. Imagine, in the
previous run X.assets.json was created and then the build failed, so that
the expected Y.assets.json was not created. In the next run {Get-ChildItem}
returns X.assets.json, its time is compared with inputs and the task is skipped.
But it should not be skipped because the expected Y.assets.json is missing.
That is why Y.assets.json (required path) must be specified in outputs.

I recall that I twitted you that outputs may be either paths or items.
Sorry, I was wrong, I should have read the help. Outputs must be paths.
(Inputs may be paths or items, this is still correct).

Please, describe your use case. I understand the inputs are all .csproj files.
It is not clear what the outputs are. I can see they are .assets.json files.
Is it a single file? Or is it one file for each project? Or something else?
I will tell you how to compose the task when I know the answer.

[nightroman]

And so far it looks like Invoke-Build throws the error correctly. By the current design, outputs are all expected output paths. The case with no outputs is not valid.

[nightroman]

If I understand your case correctly, here is one possible solution

    $RestoreInputs = Get-ChildItem -Recurse *.csproj
    $RestoreOutputs = $(
        foreach ($item in $RestoreInputs) {
            Join-Path $item.DirectoryName obj/project.assets.json
        }
    )

Then we use $RestoreInputs and $RestoreOutputs as task inputs and outputs.

Let me repeat, we cannot use anything like Get-ChildItem "..\*\obj\project.assets.json"
as output, this is wrong. If some of them exist and they are up to date then the task
would be skipped even if some needed project.assets.json are missing. Wrong.

Yes, if all project.assets.json are missing then the task must not be skipped.
In theory. But in practice, the empty output just means that a wrong expression
is used. The correct output (e.g. my version above) cannot be empty if the input
is not empty.

[daviwil]

Understood, Outputs must be statically-generated paths. However, that's what I'm doing in this example and it doesn't seem to work:

task Restore `
    -Inputs { Get-ChildItem -Recurse *.csproj } `
    -Outputs { Join-Path (Get-ChildItem .\src, .\test -Directory | Select-Object -ExpandProperty FullName) -ChildPath "obj\project.assets.json" } `
    -Before Build, BuildHost, Test {

    exec { & $script:dotnetExe restore }
}

Here's the list of file paths generated by the Outputs ScriptBlock:

PS> Join-Path (Get-ChildItem .\src, .\test -Directory | Select-Object -ExpandProperty FullName) -ChildPath "obj\project.assets.json"
C:\dev\PowerShellEditorServices\src\PowerShellEditorServices\obj\project.assets.json
C:\dev\PowerShellEditorServices\src\PowerShellEditorServices.Channel.WebSocket\obj\project.assets.json
C:\dev\PowerShellEditorServices\src\PowerShellEditorServices.Host\obj\project.assets.json
C:\dev\PowerShellEditorServices\src\PowerShellEditorServices.Protocol\obj\project.assets.json
C:\dev\PowerShellEditorServices\test\PowerShellEditorServices.Test\obj\project.assets.json
C:\dev\PowerShellEditorServices\test\PowerShellEditorServices.Test.Channel.WebSocket\obj\project.assets.json
C:\dev\PowerShellEditorServices\test\PowerShellEditorServices.Test.Host\obj\project.assets.json
C:\dev\PowerShellEditorServices\test\PowerShellEditorServices.Test.Protocol\obj\project.assets.json
C:\dev\PowerShellEditorServices\test\PowerShellEditorServices.Test.Shared\obj\project.assets.json

Though now that I look at this list, one of those files never gets created by the restore operation. That must be why the task always gets run.

Is there a way I can get verbose output to see what evaluation the incremental build check used to determine that a task should run?

[nightroman]

Is there a way I can get verbose output to see what evaluation the incremental build check used to determine that a task should run?

No. I thought of such a feature. I was not sure and I am still not sure. Every feature comes with some cost... I will think again, though.

[nightroman]

@daviwil, I am closing this issue, as far as Invoke-Build works as designed.

I have opened two new issues related to this discussion

• Incremental tasks should show some information on triggering #49 is your request, in a way. Please comment there if you have anything to add.
• Incremental Outputs block should always receive piped Inputs #50 is another idea, an improvement that may simplify coding in my example above.

[nightroman]

@daviwil
v3.2.2 outputs some more information from incremental tasks. It is minimalistic but it should be enough to catch incremental IO logic problems like you described.