8271186: Add UL option to replace newline char #4885

YaSuenag · 2021-07-23T04:09:29Z

Most of UL entries would print log in one line, however some categories (e.g. exceptions) have multiline entries as following:

[0.157s][info][exceptions] Exception <a 'java/lang/NullPointerException'{0x000000008b918f70}: test>
 thrown in interpreter method <{method} {0x00007f8335000248} 'main' '([Ljava/lang/String;)V' in 'Test'>
 at bci 9 for thread 0x00007f8330017160 (main)

It is ease to parse with log shipper (Fluent Bit, Logstash, and more) if UL can print all logs in one line.
Famous log shippers support multiline logs of course, but its configuration tends to be complex, and also some input plugins (e.g. TCP on Fluent Bit) do not support multiline logs.

So I want to introduce new UL option foldmultilines to replace all of newline char (\n) in the log entry will be replaced to 2 chars \\n. In addition, all of backslashes (\\ in C) are also replaced to 2 chars \\\\.

After this patch, we can get following logs with foldmultilines=true:

[0.166s][info][exceptions] Exception <a 'java/lang/NullPointerException'{0x000000008b918f70}: test>\n thrown in interpreter method <{method} {0x00007fbc81000248} 'main' '([Ljava/lang/String;)V' in 'Test'>\n at bci 9 for thread 0x00007fbc9c0171a0 (main)

I've also filed CSR for this issue, please review it.

Progress

Change must not contain extraneous whitespace
Commit message must refer to an issue
Change must be properly reviewed

Issue

JDK-8271186: Add UL option to replace newline char

Reviewers

Ioi Lam (@iklam - Reviewer) ⚠️ Review applies to bb3f236
David Holmes (@dholmes-ora - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/jdk pull/4885/head:pull/4885
$ git checkout pull/4885

Update a local copy of the PR:
$ git checkout pull/4885
$ git pull https://git.openjdk.java.net/jdk pull/4885/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 4885

View PR using the GUI difftool:
$ git pr show -t 4885

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/jdk/pull/4885.diff

YaSuenag · 2021-07-23T04:09:54Z

/csr needed

bridgekeeper · 2021-07-23T04:10:48Z

👋 Welcome back ysuenaga! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2021-07-23T04:12:48Z

@YaSuenag this pull request will not be integrated until the CSR request JDK-8271188 for issue JDK-8271186 has been approved.

openjdk · 2021-07-23T04:15:09Z

@YaSuenag The following label will be automatically applied to this pull request:

hotspot-runtime

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

mlbridge · 2021-07-23T08:54:22Z

Webrevs

mseledts · 2021-07-23T21:46:55Z

Could you, please, consider adding a test for this issue? Or updating an existing test?

YaSuenag · 2021-08-24T12:38:24Z

CSR has been approved, and I pushed new commit to add testcase and to update help message & manpage. Could you review?

iklam

I tested the latest version in our CI pipeline and all tests in tiers 1-2 passed.

iklam · 2021-08-24T18:20:49Z

src/hotspot/share/logging/logFileOutput.cpp

+      *equals_pos = '=';
+      success = LogFileStreamOutput::initialize(pos, errstream);
+      *equals_pos = '\0';
+      if (!success) {


I think the code can be simplified by parsing the value here:

bool fold_multilines = ....; LogFileStreamOutput::set_fold_multilines(foldmultilines);

foldmultilines is a parameter which should be applied in initialization. LogOutput and extended classes of it look like to have a rule to apply initialization parameter at initialize().
We can simplify the code as you said if we can apply foldmultiline for LogFileStreamOutput at LogFileOutput::initialize(), but I concern it breaks the semantics for UL implementation.

If I understand correctly, your concern is _fold_multilines is a property of LogFileStreamOutput, so the parsing of this option should be done inside the LogFileStreamOutput class. That way, (in the future) all subclasses of LogFileStreamOutput will be able to handle foldmultiline=true in their options by deferring to LogFileStreamOutput::initialize.

I suppose this would be useful if we have support for sockets in the future, like

-Xlog:all=socket::address=10.1.1.1,port=1234,foldmultiline=true

However, I don't think your current implementation has the right abstraction -- FoldMultilinesOptionKey is now checked both in LogFileStreamOutput::initialize() and LogFileOutput::initialize(). If we add a new LogSocketOutput in the future, we would have to duplicated the code in LogSocketOutput::initialize() as well.

I think we should defer the design of such an abstraction until we need it. Ideally, LogFileOutput and LogSocketOutput should not need to know what options are supported by their superclass, LogFileStreamOutput

For the time being, it's easier to parse all the file output options in LogFileOutput::initialize() to keep the code simple.

I don't think your current implementation has the right abstraction

Agree, I think we can (should) improve more. Its improvement is not for LogSocketStream but also for LogStd{out,err}Output.
As we discussed in CSR about stdout/err, we will work for them in next step. But they extend LogFileStreamOutput directly. So I want to refactor it before next step.

Anyway I moved foldmultiline handling into LogFileOutput in new commit.

iklam · 2021-08-24T18:20:49Z

src/hotspot/share/logging/logConfiguration.cpp

+                                         " with the character sequence."
+                                         " Escape newline (\\n) and backslash (\\) characters in the UL output"
+                                         " if it is set to true."
+                                         " Note that it works on file output only.");


How about this message?

out->print_cr(" foldmultilines=.. - If set to true, a log event that consists of multiple lines" " will be folded into a single line by escaping" " the newline (\\n) and backslash (\\) characters" " in the output.")

I think there's no need to say "it works on file output only" because this is under the heading of "Additional output-options for file outputs".

We are not "escaping" newline characters here. We went through great pains in the CSR request to describe the actual process accurately. We are replacing newlines with the sequence \ and n. We also have to replace \ with the sequence \\ so that the conversion can be reversed by a parser. Saying all this in the help output is awkward but necessary. We also need to warn that this may not be safe with character encodings other than UTF-8. So something like:

If set to true, a log event that consists of multiple lines will be folded into a single line by replacing newline characters with the sequence '\' and 'n' in the output. Existing single backslash characters will also be replaced with a sequence of two backslashes so that the conversion can be reversed. This option is safe to use with UTF-8 character encodings, but other encodings may not work.

Ok, this version is much better than mine :-)

iklam · 2021-08-24T18:20:49Z

src/java.base/share/man/java.1

+.PP
+\f[I]foldmultilines\f[R] enables to replace newline characters within
+a multiline log event with the character sequence '\\' and 'n'.
+Note that it works on file output only.


How about:

When foldmultilines is true, a log event that consists of multiple lines will be folded into a single line by escaping the newline (\n) and backslash (\) characters in the output. This option is available only for file outputs.

This needs to be expanded similar to the help info. But you can elaborate on the character encoding limitation as per the CSR request.

YaSuenag · 2021-08-25T01:58:12Z

Thanks @iklam and @dholmes-ora ! I was relieved to hear my PR passed your CI pipeline.

I updated help message and manpage. I will update foldmultilines handling after the discussion if it needs.

iklam

LGTM. Just a small nit in the man page.

iklam · 2021-08-25T05:56:40Z

src/java.base/share/man/java.1

+.RE
+.PP
+When \f[I]foldmultilines\f[R] is true, a log event that consists of multiple lines will be folded into a single line by replacing newline characters with the sequence '\\' and 'n' in the output. Existing single backslash characters will also be replaced with a sequence of two backslashes so that the conversion can be reversed. This option is safe to use with UTF-8 character encodings, but other encodings may not work. For example, it may happen inadvertently conversion in multi-byte sequences in Shift JIS and BIG5.
+This option is available only for file outputs.


How about: "For example, it may incorrectly convert multi-byte sequences in Shift JIS and BIG5."

Thanks Ioi! I updated manpage, and also I formatted the description - it was too long than other lines!

openjdk · 2021-08-25T05:58:00Z

@YaSuenag This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8271186: Add UL option to replace newline char

Reviewed-by: iklam, dholmes

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 25 new commits pushed to the master branch:

a3308af: 8272836: Limit run time for java/lang/invoke/LFCaching tests
c4c76e2: 8272811: Document the effects of building with _GNU_SOURCE in os_posix.hpp
673ce7e: 8272873: C2: Inlining should not depend on absolute call site counts
7212561: 8267188: gc/stringdedup/TestStringDeduplicationInterned.java fails with Shenandoah
e36cbd8: 8242847: G1 should not clear mark bitmaps with no marks
2ef6871: 8272447: Remove 'native' ranked Mutex
63e062f: 8236176: Parallel GC SplitInfo comment should be updated for shadow regions
c5a2712: 8272850: Drop zapping values in the Zap* option descriptions
1e3e333: 8272884: Make VoidClosure::do_void pure virtual
0f428ca: 8272570: C2: crash in PhaseCFG::global_code_motion
... and 15 more: https://git.openjdk.java.net/jdk/compare/22ef4f065315c1238216849ce9ce71b8207b43f8...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

dholmes-ora

Seems okay now.

Thanks,
David

YaSuenag · 2021-08-27T05:05:14Z

/integrate

openjdk · 2021-08-27T05:07:07Z

Going to push as commit b16a04e.
Since your change was applied there have been 34 commits pushed to the master branch:

d732c30: 8272863: Replace usages of Collections.sort with List.sort call in public java modules
fe7d708: 8272473: Parsing epoch seconds at a DST transition with a non-UTC parser is wrong
845e1ce: 8272983: G1 Add marking details to eager reclaim logging
c420530: 8272481: [macos] javax/swing/JFrame/NSTexturedJFrame/NSTexturedJFrame.java fails
e43a907: 8271315: Redo: Nimbus JTree renderer properties persist across L&F changes
11c9fd8: 8272975: ParallelGC: add documentation to heap memory layout
b94fd32: 8272859: Javadoc external links should only have feature version number in URL
9166ba3: 8272973: Incorrect compile command used by TestIllegalArrayCopyBeforeInfiniteLoop
49b2789: 8262751: RenderPipelineState assertion error in J2DDemo
a3308af: 8272836: Limit run time for java/lang/invoke/LFCaching tests
... and 24 more: https://git.openjdk.java.net/jdk/compare/22ef4f065315c1238216849ce9ce71b8207b43f8...master

Your commit was automatically rebased without conflicts.

openjdk · 2021-08-27T05:07:21Z

@YaSuenag Pushed as commit b16a04e.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

8271186: Add UL option to replace newline char

af24c9f

openjdk bot added the csr label Jul 23, 2021

openjdk bot added the hotspot-runtime label Jul 23, 2021

add inclusion of os.hpp

358fbe1

YaSuenag marked this pull request as ready for review Jul 23, 2021

openjdk bot added the rfr label Jul 23, 2021

YaSuenag added 3 commits Aug 6, 2021

Merge remote-tracking branch 'upstream/master' into JDK-8271186

189dd3b

Implement as foldmultilines

f521450

Fix test on Windows

03af43f

openjdk bot removed the csr label Aug 24, 2021

YaSuenag added 2 commits Aug 24, 2021

Merge remote-tracking branch 'upstream/master' into JDK-8271186

267ebd0

Update UL help message and manpage

55ee9cf

iklam reviewed Aug 24, 2021

View changes

Update description in help message and in manpage

19bc58c

Remove initialize function from LogFileStreamOutput

bb3f236

iklam approved these changes Aug 25, 2021

View changes

openjdk bot added the ready label Aug 25, 2021

Update manpage of java

64122a4

dholmes-ora approved these changes Aug 26, 2021

View changes

openjdk bot closed this Aug 27, 2021

openjdk bot added integrated and removed ready rfr labels Aug 27, 2021

YaSuenag deleted the JDK-8271186 branch Aug 27, 2021

YaSuenag mentioned this pull request Aug 29, 2021

8273104: Refactoring option parser for UL #5293

Closed

3 tasks

YaSuenag mentioned this pull request Sep 8, 2021

8273471: Add foldmultilines to UL for stdout/err #5407

Closed

3 tasks

8271186: Add UL option to replace newline char #4885

8271186: Add UL option to replace newline char #4885

YaSuenag commented Jul 23, 2021 •

edited by openjdk[bot] bot

YaSuenag commented Jul 23, 2021

bridgekeeper bot commented Jul 23, 2021

openjdk bot commented Jul 23, 2021

openjdk bot commented Jul 23, 2021

mlbridge bot commented Jul 23, 2021 •

edited

mseledts commented Jul 23, 2021

YaSuenag commented Aug 24, 2021

iklam left a comment

iklam Aug 24, 2021 •

edited

YaSuenag Aug 25, 2021

iklam Aug 25, 2021

YaSuenag Aug 25, 2021

iklam Aug 24, 2021

dholmes-ora Aug 25, 2021

iklam Aug 25, 2021

iklam Aug 24, 2021

dholmes-ora Aug 25, 2021

YaSuenag commented Aug 25, 2021

iklam left a comment

iklam Aug 25, 2021

YaSuenag Aug 25, 2021

openjdk bot commented Aug 25, 2021 •

edited

dholmes-ora left a comment

YaSuenag commented Aug 27, 2021

openjdk bot commented Aug 27, 2021

openjdk bot commented Aug 27, 2021

8271186: Add UL option to replace newline char #4885

8271186: Add UL option to replace newline char #4885

Conversation

YaSuenag commented Jul 23, 2021 • edited by openjdk[bot] bot

Progress

Issue

Reviewers

Reviewing

YaSuenag commented Jul 23, 2021

bridgekeeper bot commented Jul 23, 2021

openjdk bot commented Jul 23, 2021

openjdk bot commented Jul 23, 2021

mlbridge bot commented Jul 23, 2021 • edited

Webrevs

mseledts commented Jul 23, 2021

YaSuenag commented Aug 24, 2021

iklam left a comment

iklam Aug 24, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

YaSuenag commented Aug 25, 2021

iklam left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

openjdk bot commented Aug 25, 2021 • edited

dholmes-ora left a comment

YaSuenag commented Aug 27, 2021

openjdk bot commented Aug 27, 2021

openjdk bot commented Aug 27, 2021

YaSuenag commented Jul 23, 2021 •

edited by openjdk[bot] bot

mlbridge bot commented Jul 23, 2021 •

edited

iklam Aug 24, 2021 •

edited

openjdk bot commented Aug 25, 2021 •

edited