Garbled output from Gradle

[bpoland]

Hey, we're seeing very infrequently that Gradle will mangle output, I assume due to some weird control characters. Here's an example:

It seems like this could be related to some previous discussions:

• Bug with rendering characters in Gradle progress output #96 (and Buildkite timestamps are zero-width via persistent elements #97, Refactor before fixing zero-width bk APC timestamps #98, Buildkite timestamps only at start of line, via screenLine.metadata #99)
• potentially Prefixer for ANSI timestamps overzealous on escape-code-new-lines agent#1738 ?

I also found this Gradle issue where the output looks similar but I'm not sure if it's related gradle/gradle#8204

One of the more annoying parts of this issue is that when it happens, it corrupts all the output in the step/job and isn't contained to a single log group. Is there a way that terminal-to-html could prevent control characters from jumping across log groups above it somehow? It seems unlikely/unexpected that you'd want to go backwards and change log groups above.

Also interested in anyone's thoughts about what could be causing this. I have only seen it happen a few times out of hundreds of builds and I don't know what's different when it does happen...

[DrJosh9000]

Hi again @bpoland ! That console output does look funky. Have you tried gradle --console=plain as a workaround?

Is there a way that terminal-to-html could prevent control characters from jumping across log groups above it somehow? It seems unlikely/unexpected that you'd want to go backwards and change log groups above.

That's a good idea. terminal-to-html itself isn't aware of log groupings, but perhaps we could make it aware.

Also interested in anyone's thoughts about what could be causing this

Looking through the Gradle bug you linked, it could possibly be related to how the agent doesn't set a window size on the PTY. It'll need more investigating.

[bpoland]

Hi again @bpoland ! That console output does look funky. Have you tried gradle --console=plain as a workaround?

Yeah console=plain does work but the richer output is a lot more useful when it does work. It seems like a shame to turn it off just for the 1% or less of runs where the output gets mangled 😢

[DrJosh9000]

I have a theory why this is happening only very occasionally, and it relates to some other weird output in (other customers') jobs. There's a chance this is an agent bug (that I added 😣) to do with timestamp insertion.

[DrJosh9000]

I've just released v3.73.0 of the agent (and v0.11.0 of the k8s stack) with that fix - give it a whirl and let me know if you see more garbled logs!

[bpoland]

Thanks, just to follow up here -- I haven't seen any garbled output since updating, but it was also quite rare anyway :)

I'm good to close this and reopen later if I see it again?

[DrJosh9000]

Absolutely, let us know if you see it again!

[bpoland]

@DrJosh9000 I just came across a small garble in the output, much less serious than I've seen previously. This is the only time I've seen anything weird since updating. The 77% CONFIGURING [59s] and > IDLE lines are left over from when Gradle was running, and don't normally show up after it finishes.

So it's possible there is still something a little off, or could very well be a Gradle bug too haha. I don't think this is worthy of reopening this issue since it's pretty minor but just thought I'd share -- thanks again for the help here!

[bpoland]

Just to circle back here again, unfortunately we are still seeing occasional issues with Gradle output getting messed up, similar to my last screenshot :(

I think the fix here definitely improved things as it no longer seems to jump backwards and trample output from previous log groups though! So a minor issue now, but still a bit annoying...