feat(chore): Improve git.exe execution and add parallel bucket updates

[r15ch13]

Description

• By using the direct path to $appdir\git\current\mingw64\bin\git.exe this prevents spawning subprocesses and improves execution time
  • This removes calling two shims. $scoopdir\shims\git.exe calls $appdir\git\current\bin\git.exe which is also a shim.
• Fixes bucket add fails behind proxy #1615 by replacing currentuser@ with :@
• Spawns cmd.exe only if the PROXY config value is set and required by the git command
• Uses PS7 feature Foreach-Object -Parallel to run bucket updates in parallel to improve execution time
  • Not sure how to show the changelog right now, so I just added [extras] behind the commit hash
• Fallback to sequential updates for PS5
• Improve git log output

Using a fixed git.exe could be a problem if someone uses git not installed via scoop because this will not detect it with Get-Command anymore.

Motivation and Context

1. Improve updating with many buckets
2. Fixes bucket add fails behind proxy #1615

How Has This Been Tested?

• Locally tested

Checklist:

• I have ensured that I am targeting the develop branch.
• I have updated the documentation accordingly.
• I have updated the tests accordingly.
• I have added an entry in the CHANGELOG.

Before:

After:

[rasa]

This will almost certainly fix #3877 "Git spawns thousands of times".

[rasa]

Using a fixed git.exe could be a problem if someone uses git not installed via scoop because this will not detect it with Get-Command anymore.

@r15ch13 Couldn't it test to see which git is first in the path, and only implement this logic, if the first git found is scoop's shim?

[niheaven]

Using standard PowerShell argument [Switch]Log, see https://docs.microsoft.com/en-us/powershell/scripting/developer/cmdlet/activity-parameters?view=powershell-5.1

[r15ch13]

I tried using Invoke-ExternalCommand() but the redirecting of stdout breaks the output of git. E.g.:

Used Invoke-Git() code:

function Invoke-Git {
    [CmdletBinding()]
    [OutputType([String])]
    param(
        [Parameter(Mandatory = $true, Position = 0)]
        [Alias('PSPath', 'Path')]
        [ValidateNotNullOrEmpty()]
        [String]
        $WorkingDirectory,
        [Parameter(Mandatory = $true, Position = 1)]
        [Alias('Args')]
        [String[]]
        $ArgumentList
    )

    $proxy = get_config PROXY
    $git = Get-HelperPath -Helper Git

    $EnvironmentVariables = @{}
    if($proxy -and $proxy -ne 'none')  {
        if(($ArgumentList -join ' ') -Match "\b(clone|checkout|pull|fetch|ls-remote)\b") {
            # convert proxy setting for git
            if ($proxy.StartsWith('currentuser@')) {
                $proxy = $proxy.Replace("currentuser@",":@")
            }
            $EnvironmentVariables.Add('HTTPS_PROXY', $proxy)
            $EnvironmentVariables.Add('HTTP_PROXY', $proxy)
        }
    }

    Invoke-ExternalCommand -FilePath $git -WorkingDirectory $WorkingDirectory -ArgumentList $ArgumentList -EnvironmentVariables $EnvironmentVariables
}

[niheaven]

True is the output of Invoke-ExternalCommand and should be omitted by Out-Null.

And for the colorful output, a --color switch must be explicitly added to git's argument (can be done in Invoke-Git). But for --oneline, the ref inside bracket are missing (i.e., (HEAD -> master, origin/master, origin/HEAD))

PS, this is related to Invoke-Git and may be done after the PR if it needs additional tweaks (to keep consistence with former output).

[rashil2000]

Yes, the --color flag should be added

[r15ch13]

And --decorate shows the missing (HEAD -> master, origin/master, origin/HEAD).

But iec doesn't support redirecting the output to a variable which is required for getting the git commands output.

E.g.

$previousCommit = Invoke-Git -Path $currentdir -ArgumentList @('rev-parse', 'HEAD')
$currentRepo = Invoke-Git -Path $currentdir -ArgumentList @('config', 'remote.origin.url')
$currentBranch = Invoke-Git -Path $currentdir -ArgumentList 'branch'

[rashil2000]

I think we can skip on using iec for Invoke-Git for now. When iec becomes more robust, we can switch to using it.

[r15ch13]

Uses try-catch-finally to revert the environment variables.

Unified the output for both bucket update methods to:

[niheaven]

Is the @() necessary for ArgumentList? If you use comma (,), it is an [Array] for the argument already.

And I agreed that iec implementation of Invoke-Git() could be in another PR, after some refactoring of iec (Why not we called it Start-ExternalProcess()? I found it just a rewrite of Start-Process Cmdlet with more features 😄 )

[r15ch13]

Just a rebase onto the current dev branch.

[r15ch13]

Abbreviated the hash to 12 hex digits to prevent inconsistent output length.

Ready for review 😄

[niheaven]

It works for me, and two questions here:

1. Sync-Bucket is only used by Sync-Buckets once, so why not copy its code in Sync-Buckets and rename latter to Sync-Bucket?
2. Buckets name and updated time are separated by space, should updated time be aligned?

    if ($PSVersionTable.PSVersion.Major -ge 7) {
        # Parallel parameter is available since PowerShell 7
        $buckets | Where-Object { $_.valid } | ForEach-Object -ThrottleLimit 5 -Parallel {
            . "$using:PSScriptRoot\..\lib\core.ps1"

            $bucketLoc = $_.path
            $name = $_.name

            $previousCommit = Invoke-Git -Path $bucketLoc -ArgumentList @('rev-parse', 'HEAD')
            Invoke-Git -Path $bucketLoc -ArgumentList @('pull', '-q')
            if ($using:Log) {
                Invoke-Git -Path $bucketLoc -ArgumentList @('--no-pager', 'log', '--color', '--no-decorate', "--grep='^(chore)'", '--invert-grep', '--abbrev=12', "--format='tformat: * %C(yellow)%h%Creset %<|(72,trunc)%s %Cgreen$name%Creset %C(cyan)%cr%Creset'", "$previousCommit..HEAD")
            }
        }
    } else {
        $buckets | Where-Object { $_.valid } | ForEach-Object {
            $bucketLoc = $_.path
            $name = $_.name

            $previousCommit = Invoke-Git -Path $bucketLoc -ArgumentList @('rev-parse', 'HEAD')
            Invoke-Git -Path $bucketLoc -ArgumentList @('pull', '-q')
            if ($Log) {
                Invoke-Git -Path $bucketLoc -ArgumentList @('--no-pager', 'log', '--color', '--no-decorate', "--grep='^(chore)'", '--invert-grep', '--abbrev=12', "--format='tformat: * %C(yellow)%h%Creset %<|(72,trunc)%s %Cgreen$name%Creset %C(cyan)%cr%Creset'", "$previousCommit..HEAD")
            }
        }
    }

[r15ch13]

The idea for Sync-Bucket was that it just works with one bucket. For later use:tm: :smile: (whatever that may be)
And I couldn't get the parallel foreach to work with a function. So I guess merging it into Sync-Buckets is okay. 😄

Didn't want to truncate the bucket name for the alignment. But it could determine the longest name and align it.

[r15ch13]

Yes, but it doesn't update one bucket. So the name is misleading. I know PSScriptAnalyzer doesn't like it 😄
Any other name suggestions?

[niheaven]

It could update one bucket, right? Think about Rename-Item or Remove-Item, PS always uses single word for these multiple item(s) manipulating functions. So Sync-Bucket is fine IMO.

[niheaven]

LGTM!

## [Unreleased](https://github.com/ScoopInstaller/Scoop/compare/master...develop)

And the changelog? (headline above)

[rashil2000]

new output:

This output is quite wide, IMO. Default terminal emulator widths are like 30-40 columns. May I suggest outputting a PSObject list for this? (similar to scoop list)

We will lose the individual colors, but PowerShell will automatically take care of padding, alignment and terminal adjustment. (Not to mention the output will become parseable by PowerShell. Can't think of a use case for now, but who knows 😅)

[r15ch13]

Fits in both default terminals of Windows 11

This is just output for the user. If someone wants parse this it would be better to use Git directly on the buckets itself 😄

[niheaven]

@r15ch13 Occasional the pull process will be outputted, why?

[r15ch13]

hm, I don't know 🤷🏽
Never happend while testing this

[niheaven]

Tend to be appeared after a long time pull. (and maybe a poor network?)

Have seen it several times when I don't scoop update for several days.

Change LAST_UPDATE and reset bucket to weeks or months ago and run scoop update? (I'll change it and test)

[niheaven]

Wow, it is.

I've reset my buckets repo to 3 weeks ago (one or two is enough) and run scoop update, the red output appears again.

[r15ch13]

How did you reset the repo? I tried a hard reset to an older commit but it doesn't hit the network to change back to the latest.

[rashil2000]

Also the scoop bucket subcommands seem to be broken after this PR:

[r15ch13]

Can't reproduce both issues. 😕

Output of git --version?

[rashil2000]

git version 2.39.0.windows.1

[niheaven]

For specific git version?

[niheaven]

@r15ch13 If git reset gives Updating files: 100% (513/513), done. Then scoop update will output the red Updating files too.

[HUMORCE]

cannot reproduce, both Powershell 7.2 and 5.1.

git version 2.39.0.windows.1

[rashil2000]

My Scoop home is located at a non default directory, with a space in the path. Maybe that's the reason?

[niheaven]

See the Updating files prompt, @HUMORCE you haven't met them, so the output is fine.

For @rashil2000's bug, I have a non-default location of scoop w/o space, and cannot reproduce. Maybe it is the space?

[rashil2000]

Looks like all commands related to Git have an issue when Scoop home is a spaced path.