Carriage return \r being appended to the output

[lchave]

This is autogenerated. Please review and update as needed.

Describe the bug

Getting the following error when trying to query the IP address of a NIC for which I had already pulled the ID and sent as a parameter in the show command.
Operation failed with status: 'Bad Request'. Details: 400 Client Error: Bad Request for url: https://management.azure.com/subscriptions/......./resourceGroups/......./providers/Microsoft.Network/networkInterfaces/.........?api-version=2020-04-01

Command Name
az resource show

Errors:

az: error: unrecognized arguments: \ \ \

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

• Put any pre-requisite steps here...
• az resource show --name {} {} --resource-group {} {} --resource-type {} {} --query {} -o {}

Expected Behavior

Environment Summary

Windows-10-10.0.17763-SP0
Python 3.6.6
Installer: 

azure-cli 2.6.0

Extensions:
interactive 0.4.4

Additional Context

[yungezz]

hi @zhoxing-ms could you have a look? thanks

[yonzhan]

add to S171

[lchave]

Hello, just following up on this request, do you have any updates?

[zhoxing-ms]

az: error: unrecognized arguments: \ \ \

@lchave Hi, this looks like a problem with the incorrect input parameters. Could you please email me the specific parameters of the command and the printed information after adding --debug to the command? My email address is Zhou.Xing@microsoft.com , thanks~

[lchave]

@zhoxing-ms I've sent the requested information please let me know if you need anything else.

[zhoxing-ms]

@lchave Hi, through the debug log, I found that this problem was caused by the \r at the end of the passed --id parameter. It will cause the URL of the service request to be https://management.azure.com/subscriptions/{subscription-id}/resourceGroups/{resource-group}/providers/{provider}/{resource-type}/{resource-name}%0D
The %0D at the end will cause the URL to be illegal:

Bad Request - Invalid URL

---

HTTP Error 400. The request URL is invalid.

May I ask why there is a \r at the end of your resource id? Could you remove this special character?

[lchave]

@zhoxing-ms Hi! thanks for the input, we have not added the \r at the end. As you can see in the email I just sent we are only passing a variable defined in the step before it. If we echo that variable there is not a \r at the end of it.

[zhoxing-ms]

@lchave Could you please describe the environment in detail? By the way, please email me the script you used, thank you~

[arekpalinski]

I think I've run into the same issue. I followed the following example:

https://docs.microsoft.com/en-us/azure/event-grid/custom-event-to-queue-storage#subscribe-to-a-custom-topic

so I had a script like:

storageid=$(az storage account show --name $storageAccountName --resource-group $storageResourceGroup --query id --output tsv | tr -d '\r')
queueid="$storageid/queueservices/default/queues/$queueName" 

az eventgrid event-subscription create --source-resource-id $storageid --name $eventSubscriptionName --endpoint-type storagequeue --endpoint $queueid --included-event-types Microsoft.Storage.BlobCreated

what resulted in Operation failed with status: 'Bad Request'. Details: 400 Client Error: Bad Request for url: because of %0D character in the middle of URL. I have workaround this by adding | tr -d '\r' here:

storageid=$(az storage account show --name $storageAccountName --resource-group $storageResourceGroup --query id --output tsv | tr -d '\r')

[zhoxing-ms]

@arekpalinski May I ask your specific development environment? Is the WSL, Linux, Windows or others? And what is the type of this script?

[arekpalinski]

Yes, WSL. It's a bash script.

[zhoxing-ms]

@arekpalinski Hi, I have reproduced this problem in the WSL environment and identified the root cause:
The newline character in WSL is LF and the newline character in windows is CRLF. Therefore, when a script written under Windows is used in WSL, only \n in the newline character \r\n of CRLF is recognized as the newline character of LF, while \r is recognized as plain text.

For example:

1. Write a bash script in Windows, which is in CRLF format:
   The script content:

group_name=$(az group show -n zhoxing-test --query name)
echo $group_name
az group create -l westus --debug -n $group_name

2. Execute the script in the WSL environment

root@DESKTOP-S8GCCTE:/mnt/c/Users/zhoxing.FAREAST/Desktop# file test.sh
test.sh: ASCII text, with CRLF line terminators

root@DESKTOP-S8GCCTE:/mnt/c/Users/zhoxing.FAREAST/Desktop# bash test.sh
"zhoxing-test"
Command arguments: ['group', 'create', '-l', 'westus', '--debug', '-n', '"zhoxing-test"\r\r']

At this point, the value of group_name is added with \r due to incompatibility between WSL and windows newline character.

3. Convert the file format from CRLF to LF and then execute the script

root@DESKTOP-S8GCCTE:/mnt/c/Users/zhoxing.FAREAST/Desktop# dos2unix test.sh
dos2unix: converting file test.sh to Unix format...

root@DESKTOP-S8GCCTE:/mnt/c/Users/zhoxing.FAREAST/Desktop# file test.sh
test.sh: ASCII text

root@DESKTOP-S8GCCTE:/mnt/c/Users/zhoxing.FAREAST/Desktop# bash test.sh
"zhoxing-test"
Command arguments: ['group', 'create', '-l', 'westus', '--debug', '-n', '"zhoxing-test"']

Problem solving after the file format becomes LF.

For more information, please refer to: click

[zhoxing-ms]

Because there is no reply to this question for a long time, I will close it temporarily. If you have any questions, please feel free to let us know.

[jiasli]

There are 2 ways to trigger this issue on WSL.

Calling az installed on Windows

If you are calling az installed on Windows system instead of the az installed on WSL Linux, you will hit this issue.

To identify which az you are calling, run

# az installed on Windows
$ which az
/mnt/c/Program Files (x86)/Microsoft SDKs/Azure/CLI2/wbin/az

# az installed on WSL Linux
$ which az
/usr/bin/az

Call az installed on Windows, save the result and let python3 help us display its raw value:

$ result=$(az account show --query id --output tsv)
$ python3 -c "import sys; print(sys.argv)" "$result"
['-c', '0b1f6471-1bf0-4dda-aec3-cb9272f09590\r']

Solution

Always stay in WSL Linux. Make sure az is installed following Install the Azure CLI on Linux and /mnt/c/Program Files (x86)/Microsoft SDKs/Azure/CLI2/wbin/az is not called.

Now it works correctly:

$ result=$(az account show --query id --output tsv)
$ python3 -c "import sys; print(sys.argv)" "$result"
['-c', '6b085460-5f21-477e-ba44-1035046e9101']

Executing .sh script with CRLF

As @zhoxing-ms stated in #13573 (comment), say we have test.sh with CRLF as End of Line Sequence:

result="somevalue"
python3 -c "import sys; print(sys.argv)" "$result"

When executing the script in WSL:

$ . test.sh
['-c', 'somevalue\r\r']

Solution

To fix it, you may open the script with VS Code:

code test.sh

Select End of Line Sequence and change it to LF:

Now it works correctly:

$ . test.sh
['-c', 'somevalue']

[jiasli]

I have created microsoft/WSL#7248 for WSL to improve its behavior.

[treyham91]

I was also having this same issue following the samples on https://docs.microsoft.com/en-us/azure/event-grid/custom-event-to-eventhub#subscribe-to-a-custom-topic. I was able to resolve it by setting the internal field separator on the top of my bash script: IFS=$'\r\n' I'm using WSL on Windows.

[jiasli]

@treyham91, the IFS=$'\r\n' solution only works if the result (like $hubid and $topicid) is not quoted.

$ IFS=$'\r\n'
$ result=$(powershell.exe "Write-Output test")
$ python3 -c "import sys; print(sys.argv)" $result
['-c', 'test']

If an argument is quoted, Bash doesn't strip the trailing \r and still passes \r as part of the argument.

$ python3 -c "import sys; print(sys.argv)" "$result"
['-c', 'test\r']
            ^^

I would strongly recommend installing the Ubuntu distribution of Azure CLI in WSL to prevent further frustration.

For those who are interested in what $'string' is:

• https://www.gnu.org/software/bash/manual/html_node/ANSI_002dC-Quoting.html
• https://unix.stackexchange.com/questions/48106/what-does-it-mean-to-have-a-dollarsign-prefixed-string-in-a-script

[jiasli]

The /mnt/c/Program Files (x86)/Microsoft SDKs/Azure/CLI2/wbin/az (C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin\az) entry script was added by #13197. As a side effect, it now also affects WSL and causes this issue.

Actually, that entry script never worked well even in Git Bash:

$ az1 version --query '"azure-cli"' --output tsv | cat -vE
2.48.1^M$

Options we have for C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin\az:

1. Keep it, but it doesn't really work well as we will face this \r being appended to the output issue.
2. Drop it, but Git Bash or (WSL without apt-installed Azure CLI) can no longer call Azure CLI installed on Windows.

[gcrockenberg]

\r still a problem with Azure CLI v 2.49.0 on WSL Umbuntu installed using curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash from instructions on https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-linux?pivots=apt.

which az
/usr/bin/az

RESOURCE_GROUP=$(az group list --query "[0].name" -o tsv)
set
RESOURCE_GROUP=$'MyResourceGroup\r'

[jiasli]

@gcrockenberg, this is because Bash caches az's location to the Windows one before you install the DEB package. You may confirm with below commands. I am also attaching the expected result:

$ command -v az
/usr/bin/az

$ az --version
azure-cli                         2.49.0

core                              2.49.0
telemetry                          1.0.8

Dependencies:
msal                              1.20.0
azure-mgmt-resource               22.0.0

Python location '/opt/az/bin/python3'
Extensions directory '/home/user2/.azure/cliextensions'

Python (Linux) 3.10.10 (main, May 19 2023, 08:20:31) [GCC 11.3.0]

Legal docs and information: aka.ms/AzureCliLegal

The result of which az and command -v az will differ because which always returns the latest status on disk, but command -v returns the cache of the shell.

The solution is to restart your shell.