8307517: Add VMErrorCallback infrastructure to extend hs_err dumping #13824

stefank · 2023-05-05T07:57:53Z

Sometimes when we crash in the GC we'd like to get some more information about what was going on the crashing thread. One example is when Generational ZGC crashes during store barrier flushing. From https://github.com/openjdk/zgc/blob/349cf9ae38664991879402a90c5e23e291f1c1c3/src/hotspot/share/gc/z/zStoreBarrierBuffer.cpp#L245

class ZStoreBarrierBuffer::OnError : public VMErrorCallback {
private:
  ZStoreBarrierBuffer* _buffer;

public:
  OnError(ZStoreBarrierBuffer* buffer) :
      _buffer(buffer) {}

  virtual void call(outputStream* st) {
    _buffer->on_error(st);
  }
};

void ZStoreBarrierBuffer::on_error(outputStream* st) {
  st->print_cr("ZStoreBarrierBuffer: error when flushing");
  st->print_cr(" _last_processed_color: " PTR_FORMAT, _last_processed_color);
  st->print_cr(" _last_installed_color: " PTR_FORMAT, _last_installed_color);

  for (int i = current(); i < (int)_buffer_length; ++i) {
    st->print_cr(" [%2d]: base: " PTR_FORMAT " p: " PTR_FORMAT " prev: " PTR_FORMAT,
        i,
        untype(_base_pointers[i]),
        p2i(_buffer[i]._p),
        untype(_buffer[i]._prev));
  }
}

void ZStoreBarrierBuffer::flush() {
  if (!ZBufferStoreBarriers) {
    return;
  }

  OnError on_error(this);
  VMErrorCallbackMark mark(&on_error);

  for (int i = current(); i < (int)_buffer_length; ++i) {
    const ZStoreBarrierEntry& entry = _buffer[i];
    const zaddress addr = ZBarrier::make_load_good(entry._prev);
    ZBarrier::mark_and_remember(entry._p, addr);
  }

  clear();
}

If we crash in ZStoreBarrierBuffer::flush, we print the information above into the hs_err file.

We've found this information to be useful and would like to upstream the infrastructure separately from the much larger Generational ZGC PR.

Testing: this has been brewing and been used in the Generational ZGC repository for a long time.

Progress

Change must be properly reviewed (1 review required, with at least 1 Reviewer)
Change must not contain extraneous whitespace
Commit message must refer to an issue

Issue

JDK-8307517: Add VMErrorCallback infrastructure to extend hs_err dumping

Reviewers

Erik Österlund (@fisk - Reviewer)
Axel Boldt-Christmas (@xmas92 - Committer)
David Holmes (@dholmes-ora - Reviewer)
Thomas Stuefe (@tstuefe - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/13824/head:pull/13824
$ git checkout pull/13824

Update a local copy of the PR:
$ git checkout pull/13824
$ git pull https://git.openjdk.org/jdk.git pull/13824/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 13824

View PR using the GUI difftool:
$ git pr show -t 13824

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/13824.diff

Webrev

Link to Webrev Comment

bridgekeeper · 2023-05-05T07:59:10Z

👋 Welcome back stefank! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

fisk

Looks good.

openjdk · 2023-05-05T08:02:19Z

@stefank This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8307517: Add VMErrorCallback infrastructure to extend hs_err dumping

Reviewed-by: eosterlund, aboldtch, dholmes, stuefe

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 29 new commits pushed to the master branch:

ad0e5a9: 8304720: SuperWord::schedule should rebuild C2-graph from SuperWord dependency-graph
495f268: 8306966: RISC-V: Support vector cast node for Vector API
0dca573: 8301739: AArch64: Add optimized rules for vector compare with immediate for SVE
3d3eaed: 8306941: Open source several datatransfer and dnd AWT tests
1f57ce0: 8307446: RISC-V: Improve performance of floating point to integer conversion
4e4828e: 8307553: Remove dead code MetaspaceClosure::push_method_entry
7d58978: 8280031: Deprecate GTK2 for removal
b5922c3: 8305846: Support compilation in Proc test utility
73ac710: 8307425: Socket input stream read burns CPU cycles with back-to-back poll(0) calls
e2b1013: 8306326: [BACKOUT] 8277573: VmObjectAlloc is not generated by intrinsics methods which allocate objects
... and 19 more: https://git.openjdk.org/jdk/compare/1b143ba78712e7ac98ca9873c50989b3fba07394...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

openjdk · 2023-05-05T08:03:07Z

@stefank The following label will be automatically applied to this pull request:

hotspot

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

xmas92

lgtm.

I have also experimented with using this functionality to only record the last unloading events for a specific unloading cycle. (And have the callback print the events if a crash occurred during the cycle). As I found that in release builds the majority of the time is spent inside Events::log_class_unloading for do_unloading.

mlbridge · 2023-05-05T08:12:24Z

Webrevs

00: Full (5ae110e1)

dholmes-ora

This looks quite neat but I'm not clear on the need for the VMErrorCallbackMark - can't the callback link/unlink itself at construction/destruction?

tstuefe

Nice, I like it.

tstuefe · 2023-05-05T13:43:43Z

src/hotspot/share/utilities/vmError.hpp

@@ -213,4 +213,27 @@ class VMError : public AllStatic {
  static int prepare_log_file(const char* pattern, const char* default_pattern, bool overwrite_existing, char* buf, size_t buflen);

 };
+


pre-existing, can you please add a prototype decl for outputStream?

tstuefe · 2023-05-05T13:46:18Z

src/hotspot/share/utilities/vmError.hpp

+};
+
+class VMErrorCallbackMark : public StackObj {
+  Thread* _thread;


Why would we need the thread here? Why not use Thread::current in dtor? This object is only used as stack object, right?

I was treading in Runtime code and Coleen usually wants to use cached-away Thread pointers instead of calling Thread::current() repeatedly. I'm fine with either solution.

Given the context it would have to be Thread::current_or_null_safe(). But yes we prefer not to re-materialize the current thread if we already have it at hand.

Could you explain why it would have to be Thread::current_or_null_safe()? The constructor and destructor are run in "normal" JVM code and not in the error handler.

Sorry my mistake.

tstuefe · 2023-05-05T13:47:57Z

This looks quite neat but I'm not clear on the need for the VMErrorCallbackMark - can't the callback link/unlink itself at construction/destruction?

I like it better this way. Otherwise you dictate that the callback obj itself has to live on the stack. It may be large, or it may be shared between different threads.

stefank · 2023-05-05T16:40:04Z

This looks quite neat but I'm not clear on the need for the VMErrorCallbackMark - can't the callback link/unlink itself at construction/destruction?

Yes it could.

It has a couple of drawbacks, but it's unclear to me if those are important:

The linking of the callbacks happens before they have been fully constructed
It makes a strong tie between the lifecycle of the callback and the linking/unlinking. For some callbacks that might not be preferable.

The main advantage is that there's one less class and the linking-site can become a one-liner.

I can go either way, so it would be good if the reviewers could chime in with their preference. This is what it would look like:
master...stefank:jdk:8307517_VMErrorCallback_2

tstuefe · 2023-05-05T17:02:01Z

I can go either way, so it would be good if the reviewers could chime in with their preference. This is what it would look like:

master...stefank:jdk:8307517_VMErrorCallback_2

I prefer the explicit RAII object, separate from the callback.

dholmes-ora

I'm fine with the code as-is. Thanks.

stefank · 2023-05-08T07:56:00Z

Thanks for reviewing! In the interest of getting this pushed before the Generational ZGC, I'm going to integrate it now. FWIW, I'm not opposed to doing some follow-up style changes if we decide that this should be further tweaked.
/integrate

openjdk · 2023-05-08T07:58:12Z

Going to push as commit 33245d6.
Since your change was applied there have been 33 commits pushed to the master branch:

7a1cb64: 8293547: Add relaxed add_and_fetch for macos aarch64 atomics
68f385c: 8307428: jstat tests doesn't tolerate dash in the O column
8bbd264: 8307423: [s390x] Represent Registers as values
959e62c: 8307521: Introduce check_oop infrastructure to check oops in the oop class
ad0e5a9: 8304720: SuperWord::schedule should rebuild C2-graph from SuperWord dependency-graph
495f268: 8306966: RISC-V: Support vector cast node for Vector API
0dca573: 8301739: AArch64: Add optimized rules for vector compare with immediate for SVE
3d3eaed: 8306941: Open source several datatransfer and dnd AWT tests
1f57ce0: 8307446: RISC-V: Improve performance of floating point to integer conversion
4e4828e: 8307553: Remove dead code MetaspaceClosure::push_method_entry
... and 23 more: https://git.openjdk.org/jdk/compare/1b143ba78712e7ac98ca9873c50989b3fba07394...master

Your commit was automatically rebased without conflicts.

openjdk · 2023-05-08T07:58:25Z

@stefank Pushed as commit 33245d6.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

8307517: Add VMErrorCallback infrastructure to extend hs_err dumping

5ae110e

fisk approved these changes May 5, 2023

View reviewed changes

openjdk bot added ready Pull request is ready to be integrated rfr Pull request is ready for review labels May 5, 2023

openjdk bot added hotspot hotspot-dev@openjdk.org ready Pull request is ready to be integrated and removed ready Pull request is ready to be integrated labels May 5, 2023

xmas92 approved these changes May 5, 2023

View reviewed changes

openjdk bot added ready Pull request is ready to be integrated and removed ready Pull request is ready to be integrated labels May 5, 2023

dholmes-ora reviewed May 5, 2023

View reviewed changes

tstuefe approved these changes May 5, 2023

View reviewed changes

dholmes-ora approved these changes May 8, 2023

View reviewed changes

openjdk bot added the integrated Pull request has been integrated label May 8, 2023

openjdk bot closed this May 8, 2023

openjdk bot removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels May 8, 2023

		@@ -213,4 +213,27 @@ class VMError : public AllStatic {
		static int prepare_log_file(const char* pattern, const char* default_pattern, bool overwrite_existing, char* buf, size_t buflen);

		};

8307517: Add VMErrorCallback infrastructure to extend hs_err dumping #13824

8307517: Add VMErrorCallback infrastructure to extend hs_err dumping #13824

Uh oh!

Conversation

stefank commented May 5, 2023 • edited by openjdk bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Progress

Issue

Reviewers

Reviewing

Webrev

Uh oh!

bridgekeeper bot commented May 5, 2023

Uh oh!

fisk left a comment

Choose a reason for hiding this comment

Uh oh!

openjdk bot commented May 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openjdk bot commented May 5, 2023

Uh oh!

xmas92 left a comment

Choose a reason for hiding this comment

Uh oh!

mlbridge bot commented May 5, 2023

Webrevs

Uh oh!

dholmes-ora left a comment

Choose a reason for hiding this comment

Uh oh!

tstuefe left a comment

Choose a reason for hiding this comment

Uh oh!

tstuefe May 5, 2023

Choose a reason for hiding this comment

Uh oh!

tstuefe May 5, 2023

Choose a reason for hiding this comment

Uh oh!

stefank May 5, 2023

Choose a reason for hiding this comment

Uh oh!

dholmes-ora May 8, 2023

Choose a reason for hiding this comment

Uh oh!

stefank May 8, 2023

Choose a reason for hiding this comment

Uh oh!

dholmes-ora May 8, 2023

Choose a reason for hiding this comment

Uh oh!

tstuefe commented May 5, 2023

Uh oh!

stefank commented May 5, 2023

Uh oh!

tstuefe commented May 5, 2023

Uh oh!

dholmes-ora left a comment

Choose a reason for hiding this comment

Uh oh!

stefank commented May 8, 2023

Uh oh!

openjdk bot commented May 8, 2023

Uh oh!

openjdk bot commented May 8, 2023

Uh oh!

Uh oh!

stefank commented May 5, 2023 •

edited by openjdk bot

Loading

openjdk bot commented May 5, 2023 •

edited

Loading