The PowerShell Problem

[huettenhain]

In fcc36d8, I added some extremely weak PowerShell support, this was improved slightly in 504f015. The main issues are the following:

1. Don’t parse the pipeline as text when it is directed from an EXE to another EXE or file. Keep the bytes as-is. PowerShell/PowerShell#1908
2. Improve pipeline for native commands PowerShell/PowerShell#559

Our current workaround is to:

1. wrap all output as hex with a magic prefix when running under PowerShell
2. unwrap all input when we are running under PowerShell and the magic prefix is detected
3. simply accept the fact that PowerShell has no streaming stdin/stdout

If either of the above two PowerShell issues are ever resolved in a way that supports the binary refinery design, the workaround should be removed because it introduces an unnecessary encoding step.

[ImportTaste]

You could add support for Use-RawPipeline in the meantime.

[jhhcs]

Hey @ImportTaste, the way I understand Use-RawPipeline, it should already work. I have, however, not tried it out myself. If you already tested it, are there any issues to report?

[cxiao]

Doing some extremely simple tests with Use-RawPipeline, it does seem to work as expected in providing a pipeline for raw binary data.

Use-RawPipeline provides the following main cmdlets:

• Invoke-NativeCommand, aliased to run, for running a native command and treating its input and output as raw binary streams.
• Receive-RawPipeline, aliased to 2ps, for retrieving the raw binary streams from a native command pipeline, and putting it back into a form that other Powershell cmdlets can consume (optionally specifying an encoding).

There are also cmdlets to read raw binary streams from file, and dump them again back to file.

PS> run emit M7EwMzVzBkI3IwNTczM3cyMg2wQA | run b64 | run zl | run hex | 2ps
Hello World

PS> run emit "Once upon a time, at the foot of a great mountain ..." | run aes "pbkdf2[32,s4lty]:swordfish" --iv md5:X -R | run ccp md5:X | run aes "pbkdf2[32,s4lty]:swordfish" --iv cut:0:16 | 2ps
Once upon a time, at the foot of a great mountain ...

PS> run emit .\CyberChef_v9.32.3.zip | run xtzip "["| run dump "cyberchef_extracted/{path}" "]" | 2ps
PS> Get-ChildItem .\cyberchef_extracted\ -Name
assets
images
modules
ChefWorker.js.LICENSE.txt
CyberChef_v9.32.3.html
DishWorker.js.LICENSE.txt
InputWorker.js.LICENSE.txt
LoaderWorker.js.LICENSE.txt
ZipWorker.js.LICENSE.txt

Of course, certain characters which Powershell uses as part of its shell language, such as [], {}, and ,, must be quoted.

BTW, I think that my tests above bypass your PowerShell check in is_powershell_process, as I am running PowerShell 7.1.3; all PowerShell versions after 6 have the process name pwsh.exe, rather than powershell.exe which is_powershell_process checks for. Therefore, I think the tests above are not using the JSON serialization support, and it is Use-RawPipeline actually doing the work of making the streams work properly. Trying to use refinery without Use-RawPipeline definitely doesn't work at least, for me:

> emit M7EwMzVzBkI3IwNTczM3cyMg2wQA | b64 | zl | hex
(00:18:30) failure in zl: exception of type ValueError; could not detect any zlib stream.

[huettenhain]

Alright. Still not sure how to handle this in the most graceful manner. I can fix is_powershell_process to include a check for pwsh.exe, but what I would like best would be to disable the horrible JSON encoding alltogether, it only works inside a frame anyway. Then, log a warning when running inside PowerShell, unless the user is using Use-RawPipeline. However, I have no way of knowing whether that is the case as far as I can see.

As a side note, when I work with all-text input/output, I don't always have to quote [ and ] and I don't think I have fully grokked when that is the case:

PS C:\> emit Foo Bar [| ccp x | sep ]
xFoo
xBar
PS C:\> emit Foo Bar [| ccp x ]]
usage: ccp [-h] [-L] [-Q] [-0] [-v] data
ccp: error: unrecognized arguments: ]

[cxiao]

Yes, I don't think there's any way for binref units to inspect their environment in such a way that they can tell when they are being run under Use-RawPipeline's Inspect-NativeCommand.

Another way to handle this would be to display a very prominent warning as a post-install step on Windows systems spelling out the issues with piping binary data in Powershell, and either listing the workarounds (use cmd.exe, use Use-RawPipeline, or accept the current JSON-serialization-based Powershell support), or just linking users to this issue. Is it possible to do this from setup.py?

[cxiao]

Also, Powershell versions after 6 are now available on Mac and Linux, and officially supported. I think the number of people who actually use it as their day-to-day interactive shell on those platforms is extremely small, though, and the people who are doing that likely know the limitations.

[ImportTaste]

Yes, I don't think there's any way for binref units to inspect their environment in such a way that they can tell when they are being run under Use-RawPipeline's Inspect-NativeCommand.

Another way to handle this would be to display a very prominent warning as a post-install step on Windows systems spelling out the issues with piping binary data in Powershell, and either listing the workarounds (use cmd.exe, use Use-RawPipeline, or accept the current JSON-serialization-based Powershell support), or just linking users to this issue. Is it possible to do this from setup.py?

It's easy enough to tell if the module is loaded through Get-Module. If it is loaded, it just needs to run different commands.

[jhhcs]

I don't think there is any way to run Get-Module from inside Python, and I don't think there is any way for us to find out whether Use-RawPipeline is being used or not. I don't think there is a lot of benefit in supporting it; PowerShell either supports binary pipelines, or it does not - currently, it does not.

Right now I am leaning heavily towards treating a PowerShell parent process as a critical error, i.e. abort execution with a short explanation of why that is currently not supported. There are just too many ways this can go wrong unless our band-aid is completely flawless, and I am not confident I can make it so.

[jhhcs]

Alright, I had another idea about a better band-aid yesterday, and implemented it in 03436e2. This works a lot better now, and there is a warning displayed by default which can be disabled through an environment variable:

[jhhcs]

Of course, this now has the unfortunate side-effect that Use-RawPipeline does not work at all:

[ImportTaste]

Of course, this now has the unfortunate side-effect that Use-RawPipeline does not work at all:

How about an environment variable to use a Use-RawPipeline compatible method, then?

[jhhcs]

Oh yes, I like that! Not sure why I had not thought of that before.. 😅

[jhhcs]

This was implemented in 504f015. I have changed the names of the variables a bit and added information to the warning message:

(venv) PS> run emit c0019718c4d4538452affb97c70d16b7af3e4816d059010c277c4e579075c944 `
>> |run perc SETTINGS  [ `
>> |    run put keylen cut::1 `
>> |    run rc4 cut::keylen `
>> |    run xtp socket ]`
>> |2ps
WARNING: PowerShell has no support for binary pipelines or streaming. Binary Refinery uses an unreliable and slow 
workaround: It is strongly recommended to use the command processor instead. Proceed at your own peril!
- To silence this warning: $env:REFINERY_SILENCE_PS1_WARNING=1
- To disable the band-aid: $env:REFINERY_DISABLE_PS1_BANDAID=1
- To get more information: https://github.com/binref/refinery/issues/5
[BRPS1]:72656D6D2E6475636B646E732E6F72673A37303037
(venv) PS> $env:REFINERY_DISABLE_PS1_BANDAID=1
(venv) PS> run emit c0019718c4d4538452affb97c70d16b7af3e4816d059010c277c4e579075c944 `
>> |run perc SETTINGS  [ `
>> |    run put keylen cut::1 `
>> |    run rc4 cut::keylen `
>> |    run xtp socket ]`
>> |2ps
remm.duckdns.org:7007
(venv) PS>

I will push out a new release now.

[jhhcs]

There is a PR open in PowerShell that would (hopefully) fix this: PowerShell/PowerShell#17857

[huettenhain]

Alright! The PR that fixes this in PowerShell has been merged, the next PowerShell preview should support this. I am very excited to try that out.

[jhhcs]

Noting from PowerShell/PowerShell#1908 (comment) that the following command will be required to enable this experimental PowerShell feature:

Enable-ExperimentalFeature PSNativeCommandPreserveBytePipe

[huettenhain]

In PowerShell 7.4.0-preview.4, refinery now works as expected after running:

• $env:REFINERY_DISABLE_PS1_BANDAID=1
• Enable-ExperimentalFeature PSNativeCommandPreserveBytePipe

Since it will probably be quite a while before this is in an mainstream PowerShell release, I'm afraid we have to keep this issue open - but at least there is now a way to make it work inside some version of PowerShell.

[huettenhain]

Alright, this is now fully supported in PowerShell 7.4. 351e953 adds somewhat hacky code to support this, but I do consider the issue closed at this point since refinery can now work in PowerShell simply by installing the latest version.