Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

8281615: Deadlock caused by jdwp agent #7461

Closed
wants to merge 6 commits into from
Closed
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
@@ -1,5 +1,5 @@
/*
* Copyright (c) 2001, 2005, Oracle and/or its affiliates. All rights reserved.
* Copyright (c) 2001, 2022, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -81,15 +81,35 @@ cbTrackingObjectFree(jvmtiEnv* jvmti_env, jlong tag)
struct bag *
classTrack_processUnloads(JNIEnv *env)
{
debugMonitorEnter(classTrackLock);
if (deletedSignatures == NULL) {
// Class tracking not initialized, nobody's interested.
debugMonitorExit(classTrackLock);
return NULL;
}
struct bag* deleted = deletedSignatures;
deletedSignatures = bagCreateBag(sizeof(char*), 10);
debugMonitorExit(classTrackLock);
struct bag* new_bag = NULL;
struct bag* deleted = NULL;
jboolean retry = JNI_FALSE;
do {
// Avoid unnecessary allocations when class track has yet been activated.
Copy link
Contributor

@plummercj plummercj Feb 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this comment is accurate. It's not that we want to avoid unnecessary allocations. We want to avoid unnecessary tracking of loaded classes.

if (deletedSignatures != NULL) {
Copy link
Contributor

@plummercj plummercj Feb 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's now kind of hard to read what used to be the deletedSignatures == NULL case. I think you can first just check for NULL outside the lock and then return. It will make it clear that it is very low overhead in this case.

/* Allocate new bag outside classTrackLock lock to avoid deadlock.
*
* Note: jvmtiAllocate/jvmtiDeallocate() may be blocked by ongoing safepoints.
* It is dangerous to call them (via bagCreateBag/bagDestroyBag())while holding monitor(s),
* because jvmti may post events, e.g. JVMTI_EVENT_OBJECT_FREE at safepoints and event processing
* code may acquire the same monitor(s), e.g. classTrackLock in cbTrackingObjectFree(),
* which can lead to deadlock.
*/
new_bag = bagCreateBag(sizeof(char*), 10);
}
debugMonitorEnter(classTrackLock);
deleted = deletedSignatures;
if (deletedSignatures != NULL) {
if (new_bag != NULL) {
deletedSignatures = new_bag;
new_bag = NULL;
} else {
retry = JNI_TRUE;
}
}
debugMonitorExit(classTrackLock);
} while (retry == JNI_TRUE);
Copy link
Contributor

@plummercj plummercj Feb 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think the retry is necessary. Nor is the check for deletedSignatures != NULL. I assume you are trying to cover the case where initially deletedSignatures was NULL (so no bag was allocated), but is not NULL by the time you grab the lock. If you return when NULL like I suggested above, you won't have this issue. Note that if not NULL, there is no way another thread can get into this same code in clear it. This is because all callers first grab the handlerLock. Grabbing of the classTrackLock here is only done to synchronize with cbTrackingObjectFree(), not with other callers of classTrack_processUnloads().

Copy link
Contributor Author

@zhengyu123 zhengyu123 Feb 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was worried that classTrack_activate() races against classTrack_processUnloads() to set deletedSignatures, and you are right, it is also protected under handlerLock lock. It simplifies a code a lot.

bagDestroyBag(new_bag);
Copy link
Contributor

@plummercj plummercj Feb 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One other comment about classTrack_processUnloads() in general that also applies to the original version. It seems pretty inefficient. Every time the debug agent gets an event it is called. On almost ever call it is returning and empty back and allocating a new one. It seems a check for bagSize(deletedSignatures) == 0 and returning NULL if true would help performance. I also believe this can be done outside of the lock (would like David's opinion on this).

Copy link
Contributor Author

@zhengyu123 zhengyu123 Feb 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure if it is possible, but checking bagSize(deletedSignatures) == 0 seems to race against classTrack_reset() where it does not take handlerLock lock.

Copy link
Contributor

@plummercj plummercj Feb 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure if it is possible, but checking bagSize(deletedSignatures) == 0 seems to race against classTrack_reset() where it does not take handlerLock lock.

I had thought of that too, but I think the way classTrack_reset() is called, it is likely not possible for there to be a classTrack_processUnloads() also coming in because everything is shut down:

    threadControl_onDisconnect();
    standardHandlers_onDisconnect();

    /*
     * Cut off the transport immediately. This has the effect of
     * cutting off any events that the eventHelper thread might
     * be trying to send.
     */
    transport_close();
    debugMonitorDestroy(cmdQueueLock);

    /* Reset for a new connection to this VM if it's still alive */
    if ( ! gdata->vmDead ) {
        debugInit_reset(getEnv());    <--- calls classTrack_reset()
    }

Copy link
Contributor Author

@zhengyu123 zhengyu123 Feb 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But shutting down debug loop does not seem to have effect on ongoing jvmti callback, e.g. thread 4 in the bug report.

Copy link
Contributor

@plummercj plummercj Feb 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The thread 4 callback is triggered due to the debugger having requested CLASS_UNLOAD events. This is shutdown by eventHandler_reset(), which is called by debugInit_reset() before it calls classTrack_reset(). So that means by the time we get to classTrack_reset(), the thread 4 callback is no longer possible.

return deleted;
}

@@ -194,8 +214,11 @@ classTrack_initialize(JNIEnv *env)
void
classTrack_activate(JNIEnv *env)
{
// Allocate bag outside classTrackLock lock to avoid deadlock.
// See comments in classTrack_processUnloads() for details.
struct bag* new_bag = bagCreateBag(sizeof(char*), 1000);
Copy link
Member

@dholmes-ora dholmes-ora Feb 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think there can be any race during activation but this change is harmless.

Copy link
Contributor Author

@zhengyu123 zhengyu123 Feb 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure classTrack_activate() has the problem, but it is called outside of initialization, inside debug loop ... so I assume that event callback might happen.

Copy link
Contributor

@plummercj plummercj Feb 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree. This code is called when the debug agent receives a CLASS_UNLOAD EventRequest (which the debug agent handles by requesting JVMTI GC_FINISH events). So there is nothing that prevents classTrack_processUnloads() from being called at the same time.

Copy link
Member

@dholmes-ora dholmes-ora Feb 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In that case the classTrack_processUnloads is racing with the classTrack_activate call. I would not expect it to be possible to generate an event in relation to this before things have been "activated". But moving on ...

Copy link
Member

@dholmes-ora dholmes-ora Feb 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay I see from later comments that in fact no such race exists, but we have to synchronize with cbTrackingObjectFree

Copy link
Member

@dholmes-ora dholmes-ora Feb 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bah! But then a comment after that says it does exist again.

There seem to be issues understanding exactly what the concurrent call sequences can be with this code.

Copy link
Contributor

@plummercj plummercj Feb 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you might be confusing classTrack_activate() with classTrack_reset(). I believe the former has a race but the latter does not.

Copy link
Contributor

@plummercj plummercj Feb 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

...and even with the race, I still I think it is safe for a new bagSize(deletedSignatures) == 0 check to be added outside the lock. The only thing classTrack_activate() can do to deletedSignatures is set it when it is null. There is no risk of deletedSignatures being deleted and reallocated by another thread once you are in classTrack_processUnloads(). This is because it is called while holding the handlerLock, and as I explained elsewhere, by the time classTrack_reset() is called, it's no longer possible to be in classTrack_processUnloads().

debugMonitorEnter(classTrackLock);
deletedSignatures = bagCreateBag(sizeof(char*), 1000);
deletedSignatures = new_bag;
debugMonitorExit(classTrackLock);
}

@@ -213,13 +236,19 @@ cleanDeleted(void *signatureVoid, void *arg)
void
classTrack_reset(void)
{
struct bag* to_delete = NULL;
debugMonitorEnter(classTrackLock);

if (deletedSignatures != NULL) {
bagEnumerateOver(deletedSignatures, cleanDeleted, NULL);
bagDestroyBag(deletedSignatures);
to_delete = deletedSignatures;
deletedSignatures = NULL;
}

debugMonitorExit(classTrackLock);
// Deallocate bag outside classTrackLock to avoid deadlock.
// See comments in classTrack_processUnloads() for details.
if (to_delete != NULL) {
bagEnumerateOver(to_delete, cleanDeleted, NULL);
bagDestroyBag(to_delete);
}
}