[BUG] file.managed doesn't download redirected urls when behind a proxy

[v0101]

Description
As a workaround for #62949 I created the wheel file for pycurl and installed it via salt-pip. But when I try to download a file, e.g. https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.25.0/crictl-v1.25.0-linux-amd64.tar.gz i get
Error: HTTP 599: Received HTTP code 400 from proxy after CONNECT reading /kubernetes-sigs/cri-tools/releases/download/v1.25.0/crictl-v1.25.0-linux-amd64.tar.gz
This link accessed via curl gives a 302 redirect. For some reason, the redirect is not being followed. On previous version 3004.2 (non-onedir) this worked.

Setup
(Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)

Please be as specific as possible and give set-up details.

• on-prem machine
• VM (Virtualbox, KVM, etc. please specify)
• VM running on a cloud service, please be explicit and add details
• container (Kubernetes, Docker, containerd, etc. please specify)
• or a combination, please be explicit
• jails if it is FreeBSD
• classic packaging
• onedir packaging
• used bootstrap to install

Steps to Reproduce the behavior
I use Almalinux 9, master is 3004.2 and minion is 3005.1

dnf install libcurl-devel gcc python3-devel openssl-devel python3-pip
pip3 install wheel
pip3 wheel pycurl
salt-pip install tornado
salt-pip install pycurl-7.45.1-cp39-cp39-linux_x86_64.whl

In the state file i then try to download the file:

crictl_source:
  file.managed:
    - name: /usr/local/tmp/crictl-1.25.0-linux-amd64.tar.gz
    - source: https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.25.0/crictl-v1.25.0-linux-amd64.tar.gz
    - source_hash: 86ab210c007f521ac4cdcbcf0ae3fb2e10923e65f16de83e0e1db191a07f0235
    - user: root
    - group: root
    - mode: 644
    - makedirs: true

Expected behavior
The file is downloaded.

Versions Report

master # salt --versions-report
Salt Version:
          Salt: 3004.2
 
Dependency Versions:
          cffi: 1.11.5
      cherrypy: Not Installed
      dateutil: 2.6.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.10.1
       libgit2: Not Installed
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.14
      pycrypto: Not Installed
  pycryptodome: Not Installed
        pygit2: Not Installed
        Python: 3.6.8 (default, Sep 13 2022, 07:19:15)
  python-gnupg: Not Installed
        PyYAML: 3.12
         PyZMQ: 19.0.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: almalinux 8.6 Sky Tiger
        locale: UTF-8
       machine: x86_64
       release: 4.18.0-372.26.1.el8_6.x86_64
        system: Linux
       version: AlmaLinux 8.6 Sky Tiger

minion # salt-minion --versions-report
Salt Version:
          Salt: 3005.1
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.0
       libgit2: Not Installed
      M2Crypto: 0.38.0
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
        Python: 3.9.15 (main, Nov  1 2022, 00:00:00)
  python-gnupg: 0.4.8
        PyYAML: 5.4.1
         PyZMQ: 23.2.0
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: almalinux 9.0 Emerald Puma
        locale: utf-8
       machine: x86_64
       release: 5.14.0-70.30.1.el9_0.x86_64
        system: Linux
       version: AlmaLinux 9.0 Emerald Puma
 

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[OrangeDog]

It does follow redirects, but you are not being given one.

Salt is receiving a 599, with a report that a proxy has returned a 400. There's no 302 in any of that.

Check the proxy settings that you're giving to Salt, as that's probably where the issue is.
Or possibly with how you've configured the pycurl you built.
Sometimes GitHub also exceeds its Azure quota, causing release downloads to fail for a bit, but I forget which response code you get for that.

[v0101]

The 302 i get when i run curl -I https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.25.0/crictl-v1.25.0-linux-amd64.tar.gz
The proxy settings are the same in /etc/salt/minion.d/00-proxy.conf and in the environment for curl. So I am a bit confused about the 599.

pycurl was built by pip, so I did not set any special configuration for it.

I also tried it with https://dl.k8s.io/release/v1.24.4/bin/linux/amd64/kubelet so I don't think azure is the issue here.

[v0101]

I did a tcpdump of both the curl command line and the salt run. In both i get the 302 redirect:
curl

Frame 33: 1886 bytes on wire (15088 bits), 1886 bytes captured (15088 bits)
Linux cooked capture v2
Internet Protocol Version 4, Src: 172.30.240.100, Dst: 172.30.241.232
Transmission Control Protocol, Src Port: 8080, Dst Port: 48966, Seq: 4115, Ack: 975, Len: 1826
    Source Port: 8080
    Destination Port: 48966
    [Stream index: 0]
    [Conversation completeness: Complete, WITH_DATA (31)]
    [TCP Segment Len: 1826]
    Sequence Number: 4115    (relative sequence number)
    Sequence Number (raw): 153878155
    [Next Sequence Number: 5941    (relative sequence number)]
    Acknowledgment Number: 975    (relative ack number)
    Acknowledgment number (raw): 2486490739
    0101 .... = Header Length: 20 bytes (5)
    Flags: 0x018 (PSH, ACK)
    Window: 62
    [Calculated window size: 31744]
    [Window size scaling factor: 512]
    Checksum: 0x41c7 [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    [Timestamps]
    [SEQ/ACK analysis]
    TCP payload (1826 bytes)
    TCP segment data (368 bytes)
    [PDU Size: 2784]
[2 Reassembled TCP Segments (1392 bytes): #31(1024), #33(368)]
    [Frame: 31, payload: 0-1023 (1024 bytes)]
    [Frame: 33, payload: 1024-1391 (368 bytes)]
    [Segment count: 2]
    [Reassembled TCP length: 1392]
    [Reassembled TCP Data: 170303056b21223140a3af478f211e30dea19f500ed8db3d43d59769e2e62f60d7049087…]
Hypertext Transfer Protocol
    [Proxy-Connect-Hostname: github.com]
    [Proxy-Connect-Port: 443]
Transport Layer Security
    TLSv1.3 Record Layer: Application Data Protocol: HyperText Transfer Protocol 2
        Opaque Type: Application Data (23)
        Version: TLS 1.2 (0x0303)
        Length: 1387
        [Content Type: Application Data (23)]
        Encrypted Application Data: 21223140a3af478f211e30dea19f500ed8db3d43d59769e2e62f60d7049087f514d212f4…
        [Application Data Protocol: HyperText Transfer Protocol 2]
    TLS segment data (1370 bytes)
Hypertext Transfer Protocol
    [Proxy-Connect-Hostname: github.com]
    [Proxy-Connect-Port: 443]
Transport Layer Security
    TLSv1.3 Record Layer: Application Data Protocol: HyperText Transfer Protocol 2
        Opaque Type: Application Data (23)
        Version: TLS 1.2 (0x0303)
        Length: 1387
        [Content Type: Application Data (23)]
        Encrypted Application Data: cdfc17be98adb86e46f957ca824435ea3ce0b0f8c21e791730d9d4ae1c9b015811f5b169…
        [Application Data Protocol: HyperText Transfer Protocol 2]
    TLS segment data (1370 bytes)
    TLSv1.3 Record Layer: Application Data Protocol: HyperText Transfer Protocol 2
        Opaque Type: Application Data (23)
        Version: TLS 1.2 (0x0303)
        Length: 61
        [Content Type: Application Data (23)]
        Encrypted Application Data: 41ba60060b6c1b5a4bc994feae10a4fc28a73093c04d20e16dad7798d68ddef92462594c…
        [Application Data Protocol: HyperText Transfer Protocol 2]
    TLS segment data (44 bytes)
[3 Reassembled TLS segments (2784 bytes): #33(1370), #33(1370), #33(44)]
HyperText Transfer Protocol 2
    Stream: HEADERS, Stream ID: 1, Length 2775, 302 Found
        Length: 2775
        Type: HEADERS (1)
        Flags: 0x05, End Headers, End Stream
        0... .... .... .... .... .... .... .... = Reserved: 0x0
        .000 0000 0000 0000 0000 0000 0000 0001 = Stream Identifier: 1
        [Pad Length: 0]
        Header Block Fragment: 4803333032760a4769744875622e636f6d611d5475652c20303620446563203230323220…
        [Header Length: 2948]
        [Header Count: 15]
        Header: :status: 302 Found
        Header: server: GitHub.com
        Header: date: Tue, 06 Dec 2022 15:06:50 GMT
        Header: content-type: text/html; charset=utf-8
        Header: vary: X-PJAX, X-PJAX-Container, Turbo-Visit, Turbo-Frame, Accept-Encoding, Accept, X-Requested-With
        Header: location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/80172100/6a7ce7c5-7c3f-4b2b-b0b0-fe5fb498a36c?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20221206%2Fus-east-1%2Fs3
        Header: cache-control: no-cache
        Header: strict-transport-security: max-age=31536000; includeSubdomains; preload
        Header: x-frame-options: deny
        Header: x-content-type-options: nosniff
        Header: x-xss-protection: 0
        Header: referrer-policy: no-referrer-when-downgrade
        Header: content-security-policy: default-src 'none'; base-uri 'self'; block-all-mixed-content; child-src github.com/assets-cdn/worker/ gist.github.com/assets-cdn/worker/; connect-src 'self' uploads.github.com objects-origin.githubuserconte
        Header: content-length: 0
        Header: x-github-request-id: F275:11E8:D86B98D:DF88AE1:638F5A89

salt

Frame 32: 426 bytes on wire (3408 bits), 426 bytes captured (3408 bits)
Linux cooked capture v2
Internet Protocol Version 4, Src: 172.30.240.100, Dst: 172.30.241.232
Transmission Control Protocol, Src Port: 8080, Dst Port: 55210, Seq: 5554, Ack: 1012, Len: 366
    Source Port: 8080
    Destination Port: 55210
    [Stream index: 0]
    [Conversation completeness: Complete, WITH_DATA (31)]
    [TCP Segment Len: 366]
    Sequence Number: 5554    (relative sequence number)
    Sequence Number (raw): 3311262161
    [Next Sequence Number: 5920    (relative sequence number)]
    Acknowledgment Number: 1012    (relative ack number)
    Acknowledgment number (raw): 1770980370
    0101 .... = Header Length: 20 bytes (5)
    Flags: 0x018 (PSH, ACK)
    Window: 64
    [Calculated window size: 32768]
    [Window size scaling factor: 512]
    Checksum: 0xd8f0 [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    [Timestamps]
    [SEQ/ACK analysis]
    TCP payload (366 bytes)
    TCP segment data (300 bytes)
    [PDU Size: 2784]
[2 Reassembled TCP Segments (1392 bytes): #30(1092), #32(300)]
    [Frame: 30, payload: 0-1091 (1092 bytes)]
    [Frame: 32, payload: 1092-1391 (300 bytes)]
    [Segment count: 2]
    [Reassembled TCP length: 1392]
    [Reassembled TCP Data: 170303056beb18f14112b837aad576cd505e1fdfef5a1e55d9dbcd994c0ce15cd6d4d83c…]
Hypertext Transfer Protocol
    [Proxy-Connect-Hostname: github.com]
    [Proxy-Connect-Port: 443]
Transport Layer Security
    TLSv1.3 Record Layer: Application Data Protocol: HyperText Transfer Protocol 2
        Opaque Type: Application Data (23)
        Version: TLS 1.2 (0x0303)
        Length: 1387
        [Content Type: Application Data (23)]
        Encrypted Application Data: eb18f14112b837aad576cd505e1fdfef5a1e55d9dbcd994c0ce15cd6d4d83c8ab1360acf…
        [Application Data Protocol: HyperText Transfer Protocol 2]
    TLS segment data (1370 bytes)
Hypertext Transfer Protocol
    [Proxy-Connect-Hostname: github.com]
    [Proxy-Connect-Port: 443]
Transport Layer Security
    TLSv1.3 Record Layer: Application Data Protocol: HyperText Transfer Protocol 2
        Opaque Type: Application Data (23)
        Version: TLS 1.2 (0x0303)
        Length: 61
        [Content Type: Application Data (23)]
        Encrypted Application Data: c779e280904524172801e5a7146e4cfd9bb551d1f48ecee463df6948fe5d7f744d859b48…
        [Application Data Protocol: HyperText Transfer Protocol 2]
    TLS segment data (44 bytes)
[3 Reassembled TLS segments (2784 bytes): #30(1370), #32(1370), #32(44)]
HyperText Transfer Protocol 2
    Stream: HEADERS, Stream ID: 1, Length 2775, 302 Found
        Length: 2775
        Type: HEADERS (1)
        Flags: 0x05, End Headers, End Stream
        0... .... .... .... .... .... .... .... = Reserved: 0x0
        .000 0000 0000 0000 0000 0000 0000 0001 = Stream Identifier: 1
        [Pad Length: 0]
        Header Block Fragment: 4803333032760a4769744875622e636f6d611d5475652c20303620446563203230323220…
        [Header Length: 2948]
        [Header Count: 15]
        Header: :status: 302 Found
        Header: server: GitHub.com
        Header: date: Tue, 06 Dec 2022 14:46:58 GMT
        Header: content-type: text/html; charset=utf-8
        Header: vary: X-PJAX, X-PJAX-Container, Turbo-Visit, Turbo-Frame, Accept-Encoding, Accept, X-Requested-With
        Header: location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/80172100/6a7ce7c5-7c3f-4b2b-b0b0-fe5fb498a36c?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20221206%2Fus-east-1%2Fs3
        Header: cache-control: no-cache
        Header: strict-transport-security: max-age=31536000; includeSubdomains; preload
        Header: x-frame-options: deny
        Header: x-content-type-options: nosniff
        Header: x-xss-protection: 0
        Header: referrer-policy: no-referrer-when-downgrade
        Header: content-security-policy: default-src 'none'; base-uri 'self'; block-all-mixed-content; child-src github.com/assets-cdn/worker/ gist.github.com/assets-cdn/worker/; connect-src 'self' uploads.github.com objects-origin.githubuserconte
        Header: content-length: 0
        Header: x-github-request-id: E9FF:4EDE:D097418:D77CB4A:638F561B

But the next request fails with salt.
Redirected curl

Frame 38: 210 bytes on wire (1680 bits), 210 bytes captured (1680 bits)
Linux cooked capture v2
Internet Protocol Version 4, Src: 172.30.241.232, Dst: 172.30.240.100
Transmission Control Protocol, Src Port: 48968, Dst Port: 8080, Seq: 1, Ack: 1, Len: 150
    Source Port: 48968
    Destination Port: 8080
    [Stream index: 2]
    [Conversation completeness: Complete, WITH_DATA (31)]
    [TCP Segment Len: 150]
    Sequence Number: 1    (relative sequence number)
    Sequence Number (raw): 2498721712
    [Next Sequence Number: 151    (relative sequence number)]
    Acknowledgment Number: 1    (relative ack number)
    Acknowledgment number (raw): 814931773
    0101 .... = Header Length: 20 bytes (5)
    Flags: 0x018 (PSH, ACK)
    Window: 502
    [Calculated window size: 64256]
    [Window size scaling factor: 128]
    Checksum: 0x3b3b [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    [Timestamps]
    [SEQ/ACK analysis]
    TCP payload (150 bytes)
Hypertext Transfer Protocol
    CONNECT objects.githubusercontent.com:443 HTTP/1.1\r\n
        [Expert Info (Chat/Sequence): CONNECT objects.githubusercontent.com:443 HTTP/1.1\r\n]
            [CONNECT objects.githubusercontent.com:443 HTTP/1.1\r\n]
            [Severity level: Chat]
            [Group: Sequence]
        Request Method: CONNECT
        Request URI: objects.githubusercontent.com:443
        Request Version: HTTP/1.1
    Host: objects.githubusercontent.com:443\r\n
    User-Agent: curl/7.76.1\r\n
    Proxy-Connection: Keep-Alive\r\n
    \r\n
    [Full request URI: objects.githubusercontent.com:443]
    [HTTP request 1/1]
    [Response in frame: 40]

Redirected salt

Frame 37: 190 bytes on wire (1520 bits), 190 bytes captured (1520 bits)
Linux cooked capture v2
Internet Protocol Version 4, Src: 172.30.241.232, Dst: 172.30.240.100
Transmission Control Protocol, Src Port: 55214, Dst Port: 8080, Seq: 1, Ack: 1, Len: 130
    Source Port: 55214
    Destination Port: 8080
    [Stream index: 2]
    [Conversation completeness: Complete, WITH_DATA (63)]
    [TCP Segment Len: 130]
    Sequence Number: 1    (relative sequence number)
    Sequence Number (raw): 3937199183
    [Next Sequence Number: 131    (relative sequence number)]
    Acknowledgment Number: 1    (relative ack number)
    Acknowledgment number (raw): 2294597533
    0101 .... = Header Length: 20 bytes (5)
    Flags: 0x018 (PSH, ACK)
    Window: 502
    [Calculated window size: 64256]
    [Window size scaling factor: 128]
    Checksum: 0x3b27 [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    [Timestamps]
    [SEQ/ACK analysis]
    TCP payload (130 bytes)
Hypertext Transfer Protocol
    CONNECT objects.githubusercontent.com:443 HTTP/1.1\r\n
        [Expert Info (Chat/Sequence): CONNECT objects.githubusercontent.com:443 HTTP/1.1\r\n]
            [CONNECT objects.githubusercontent.com:443 HTTP/1.1\r\n]
            [Severity level: Chat]
            [Group: Sequence]
        Request Method: CONNECT
        Request URI: objects.githubusercontent.com:443
        Request Version: HTTP/1.1
    Host: objects.githubusercontent.com:443\r\n
    (nil)Proxy-Connection: Keep-Alive\r\n
        [Expert Info (Warning/Protocol): Illegal characters found in header name]
            [Illegal characters found in header name]
            [Severity level: Warning]
            [Group: Protocol]
    \r\n
    [Full request URI: objects.githubusercontent.com:443]
    [HTTP request 1/1]
    [Response in frame: 38]

It looks like something is wrong with the Proxy-Connection header is wrong.

The response is then the 400

Frame 38: 247 bytes on wire (1976 bits), 247 bytes captured (1976 bits)
Linux cooked capture v2
Internet Protocol Version 4, Src: 172.30.240.100, Dst: 172.30.241.232
Transmission Control Protocol, Src Port: 8080, Dst Port: 55214, Seq: 1, Ack: 131, Len: 187
    Source Port: 8080
    Destination Port: 55214
    [Stream index: 2]
    [Conversation completeness: Complete, WITH_DATA (63)]
    [TCP Segment Len: 187]
    Sequence Number: 1    (relative sequence number)
    Sequence Number (raw): 2294597533
    [Next Sequence Number: 189    (relative sequence number)]
    Acknowledgment Number: 131    (relative ack number)
    Acknowledgment number (raw): 3937199313
    0101 .... = Header Length: 20 bytes (5)
    Flags: 0x011 (FIN, ACK)
    Window: 60
    [Calculated window size: 30720]
    [Window size scaling factor: 512]
    Checksum: 0x42d6 [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    [Timestamps]
    [SEQ/ACK analysis]
    TCP payload (187 bytes)
Hypertext Transfer Protocol
    HTTP/1.0 400 Bad request\r\n
        [Expert Info (Chat/Sequence): HTTP/1.0 400 Bad request\r\n]
            [HTTP/1.0 400 Bad request\r\n]
            [Severity level: Chat]
            [Group: Sequence]
        Response Version: HTTP/1.0
        Status Code: 400
        [Status Code Description: Bad Request]
        Response Phrase: Bad request
    Cache-Control: no-cache\r\n
    Connection: close\r\n
    Content-Type: text/html\r\n
    \r\n
    [HTTP response 1/1]
    [Time since request: 0.000507000 seconds]
    [Request in frame: 37]
    [Request URI: objects.githubusercontent.com:443]
    File Data: 90 bytes
Line-based text data: text/html (3 lines)

In the salt-minion log file i see the following errors:

2022-12-06 16:11:32,025 [salt._logging.impl:1069][ERROR   ][19334] An un-handled exception was caught by Salt's global exception handler:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in position 3: invalid start byte
Traceback (most recent call last):
  File "salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in position 3: invalid start byte
2022-12-06 16:11:32,054 [salt._logging.impl:1069][ERROR   ][19334] An un-handled exception was caught by Salt's global exception handler:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbf in position 6: invalid start byte
Traceback (most recent call last):
  File "salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbf in position 6: invalid start byte
2022-12-06 16:11:32,055 [salt._logging.impl:1069][ERROR   ][19334] An un-handled exception was caught by Salt's global exception handler:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb1 in position 3: invalid start byte
Traceback (most recent call last):
  File "salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb1 in position 3: invalid start byte
2022-12-06 16:11:32,057 [salt._logging.impl:1069][ERROR   ][19334] An un-handled exception was caught by Salt's global exception handler:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x85 in position 13: invalid start byte
Traceback (most recent call last):
  File "salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x85 in position 13: invalid start byte
2022-12-06 16:11:32,058 [salt._logging.impl:1069][ERROR   ][19334] An un-handled exception was caught by Salt's global exception handler:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd8 in position 9: invalid continuation byte
Traceback (most recent call last):
  File "salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd8 in position 9: invalid continuation byte
2022-12-06 16:11:32,059 [salt._logging.impl:1069][ERROR   ][19334] An un-handled exception was caught by Salt's global exception handler:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xdd in position 5: invalid continuation byte
Traceback (most recent call last):
  File "salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xdd in position 5: invalid continuation byte
2022-12-06 16:11:32,065 [salt._logging.impl:1069][ERROR   ][19334] An un-handled exception was caught by Salt's global exception handler:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa3 in position 6: invalid start byte
Traceback (most recent call last):
  File "salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa3 in position 6: invalid start byte
2022-12-06 16:11:32,066 [salt._logging.impl:1069][ERROR   ][19334] An un-handled exception was caught by Salt's global exception handler:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa3 in position 6: invalid start byte
Traceback (most recent call last):
  File "salt/ext/tornado/curl_httpclient.py", line 497, in _curl_debug
    debug_msg = native_str(debug_msg)
  File "salt/ext/tornado/escape.py", line 219, in to_unicode
    return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa3 in position 6: invalid start byte

[OrangeDog]

Ok, so it did follow the redirect, so that's not the problem.

It appears something has inserted a NUL byte into the headers of Salt's request to your proxy, which would correctly result in a 400 error.

[v0101]

Since salt-pip install pycurl would install the same version, I don't think that fixing #62949 will fix this issue.
I updated the title.

[OrangeDog]

The same version of pycurl, but not necessarily built with the same version or configuration of libcurl and its other dependencies.

[v0101]

Ok, I see. So I guess I have to wait for #62949 to be fixed?

[OrangeDog]

Or details of what your proxy is and how it's configured so it might be possible to reproduce.

[v0101]

The version we are using is squid 4.12.
What details about the configuration would you need?

[pokerstars-gyankov]

We also experience this, any news on possible fix?