Request for improved handling of decode errors during builds.

[Cptmorgan27]

Hi,

Recently, my team encountered an error during a docker build where we hit the docker build log limit, and the output was clipped; this resulted in a random non-utf-8 character being produced. The log limit non-deterministically clipped off content in the middle of a UTF-8 Character. In our logs, there was a UTF-8 Character ➤ present; this is made up of 3 bytes (see Find all Unicode Characters from Hieroglyphs to Dingbats – Unicode Compart ) 0xE2 0x9E 0xA4. the 1MiB log clipping terminates the output at 0xE2 0x9E. It took us some time and troubleshooting to come to this finding.

We are asking if there could be more graceful handling of these decoding errors implemented as we believe that the current handling covered up the log limit error and produced a stack trace with the not-so-helpful UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 10-11: invalid continuation byte

Thank you!

[gabrieldemarmiesse]

Thank you for the issue! Can you tell me how you called buildx exactly? which arguments in the function were used I mean. I'm trying to understand better the situation. Also since it happened to you, what do you think would be a good error message to display to the user in this situation? Thanks!

[LewisGaul]

It might be better to use one of the other decode error handling options, e.g. ignore errors instead of raising https://docs.python.org/3/library/codecs.html#error-handlers. Or could explicitly handle cases where decode errors are 'expected' by checking it occurs at the end of the string and just trimming the end off.

[Cptmorgan27]

We were calling python_on_whales.docker.build as such

python_on_whales.docker.build(
                context_path=Path.cwd(),
                secrets=secrets,
                stream_logs=True,
                **docker_build_params.get_denormalized_params(...)
               )

Note get_denormalized_params() is a function we have that maps common build params for python_on_whales.docker.build or docker-py.build, which ever is being used; this is in a common use internal library we have.

We are currently working on adding builder="larger_log" to the function call to help with the build log limit.

But yes, something along @LewisGaul comment would do. Mainly a way that allows for the docker build error `[output clipped, log limit 1MiB reached] to be raised to the user.

[gabrieldemarmiesse]

Thank you for the feedback! It's much clearer. A pull request is welcomed if someone is motivated, otherwise I'll implement it when I have the time

[Cptmorgan27]

@gabrieldemarmiesse my colleague has opened a PR to address this issue please review at your earliest convenience.

[gabrieldemarmiesse]

Great, thank you :)

[magichair]

@gabrieldemarmiesse Happy to get a new release cut whenever you're ready and we can test it out over here. 🙏 Thank you.

[gabrieldemarmiesse]

I'm planning to do it on Thursday :)

[gabrieldemarmiesse]

The new release is here, can you try it and tell me if that works for you? If yes, then we can close this issue

[gabrieldemarmiesse]

closing this issue as it seems resolved