Base64decode(x).decode("utf-8") doesn't handle multiple escape characters properly

[funkybunchofoats]

Bug report

if there are multiple escape sequences back to back, the .decode(x) may "eat" trailing escape sequences/special characters.
base64.b64decode(x).decode('utf-8')
for example, when parsing HL7 messages from base64 encoding I had the following case:
b"\\.br\\\n"
this should decode to: "\.br\\n"
but instead is translated to "\.br"

I was able to work around by doing the decode in two steps and using a replacement of b"\\.br\\\n" to b"\n\n" as that what the decoded message should look like.

Your environment

• CPython versions tested on: 3.10.7
• Operating system and architecture: Windows Server 2016 64 bit

[pochmann]

When I try base64.b64decode(x).decode('utf-8') with x = b"\\.br\\\n", I get an error. Are you sure that's what you're doing?

[funkybunchofoats]

Sorry, that's after it's been b64decode but before decode.
The original HL7 message has patient info so I can't paste it here.

the binary string after base64 decode is correct. it's the second decode that's the problem.

the commented line creates the bad string.
the two uncommented lines are my workaround.

in short the base64.b64decode(x) works perfectly.
it's the decode from the binary string that's broken

[funkybunchofoats]

if you went:

x=b'\\.br\\\n'
y=base64.b64encode(x)
y2=base64.b64decode(y).decode('utf-8')

you would see the problem

[pochmann]

If the b64decode works perfectly, then why mention it? It's a distraction then.

Your

x=b'\\.br\\\n'
y=base64.b64encode(x)
y2=base64.b64decode(x).decode('utf-8')

does the exact same thing, except it creates the y that you're not using.

[funkybunchofoats]

If the b64decode works perfectly, then why mention it? It's a distraction then.

Your

x=b'\\.br\\\n'
y=base64.b64encode(x)
y2=base64.b64decode(x).decode('utf-8')

does the exact same thing, except it creates the y that you're not using.

I fixed the typo.
y2 will be the problem string not equivalent to the binary one encoded.

[pochmann]

When I run the updated version, I do not see the problem. I don't see anything, as there's no print. If I print y2, I still don't see the problem, it's the expected correct result. Unclear what you're looking at (since you're not printing).

(I didn't check with the exact same Python version, but I highly doubt yours is buggy like that, more likely you made another mistake.)

[pochmann]

What output do you get when you run this program?

import base64

x=b'\\.br\\\n'
y=base64.b64encode(x)
y2=base64.b64decode(y).decode('utf-8')
print(repr(y2))

[funkybunchofoats]

What output do you get when you run this program?

import base64

x=b'\\.br\\\n'
y=base64.b64encode(x)
y2=base64.b64decode(y).decode('utf-8')
print(repr(y2))

print(repr(y2)) gives me the input string I expect.
and print(y2) gives me the output I expect. I anticipated that would show the issue however it does not.

I don't understand why the full HL7 message breaks in this specific segment. I'll see if I can clean the segment of patient data and then upload it.

[funkybunchofoats]

Sorry, it looks like the problem is with the SQL Server ODBC driver, not the base64 package in Python.
Switching to SQL Native Client driver and the problem disappears.
My sincere apologies for wasting your time.

Essentially, I was applying the decode in a dataframe and then immediately putting into the SQL server.
After following the steps above and seeing the problem wasn't with the decoded string, it was with the insert to SQL server, I tried using the Native Client driver instead of the SQL server driver and the problem is non existent.