Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

8281615: Deadlock caused by jdwp agent #7461

Closed
wants to merge 6 commits into from
Closed
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
@@ -1,5 +1,5 @@
/*
* Copyright (c) 2001, 2005, Oracle and/or its affiliates. All rights reserved.
* Copyright (c) 2001, 2022, Oracle and/or its affiliates. All rights reserved.
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
*
* This code is free software; you can redistribute it and/or modify it
@@ -56,6 +56,14 @@ struct bag* deletedSignatures;
*/
static jrawMonitorID classTrackLock;

/*
* Note: jvmtiAllocate/jvmtiDeallocate() may be blocked by ongoing safepoints.
* It is dangerous to call them (via bagCreateBag/bagDestroyBag())while holding monitor(s),
* because jvmti may post events, e.g. JVMTI_EVENT_OBJECT_FREE at safepoints and its event
* handler may acquire the same monitor(s), e.g. classTrackLock in cbTrackingObjectFree(),
* which can lead to deadlock.
*/
Copy link
Contributor

@plummercj plummercj Feb 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The debug agent does a lot alloc/dealloc calls while processing JVMTI events and holding locks. So the question is why this is not problematic other than this classTrackLock issue. I believe the answer is that cbTrackingObjectFree() is unique in that it is handled outside of the debug agent's normal event handling code, which is serialized. cbTrackingObjectFree() is called so the debug agent can maintain a list of loaded classes, and is not an event that gets passed on to the debugger like most JVMTI events are. So we have a case here where classTrackLock can be grabbed by both "typical" JVMTI event handling code via the classTrack_processUnloads() call, and then this special cbTrackingObjectFree() event handling.

I think having this comment here doesn't help the reader of the code below unless they somehow read the comment first and then recognized it's application in the code below. At the very least the code below should tersely state while the lock is being released, and then refer to this comment for details.

Copy link
Contributor Author

@zhengyu123 zhengyu123 Feb 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I moved comments inside classTrack_processUnloads(), let me know if you prefer other places.


/*
* Invoke the callback when classes are freed, find and record the signature
* in deletedSignatures. Those are only used in addPreparedClass() by the
@@ -88,8 +96,18 @@ classTrack_processUnloads(JNIEnv *env)
return NULL;
}
struct bag* deleted = deletedSignatures;
deletedSignatures = bagCreateBag(sizeof(char*), 10);
deletedSignatures = NULL;
debugMonitorExit(classTrackLock);
Copy link
Member

@dholmes-ora dholmes-ora Feb 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks risky as the critical section is broken and the NULL deleted signatures exposed. If cbTrackingObjectFree occurs while this is true then you will lose the record of the deleted signature.

Alternatively you could allow for lock-free reading of deletedSignatures, preemptively allocate a new bad if needed then take the lock. Or even use the lock to read deletedSignatures to determine if a new bag is needed, then drop the lock, create the bag, take the lock and re-check everything.

Copy link
Contributor

@plummercj plummercj Feb 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed.

Copy link
Contributor Author

@zhengyu123 zhengyu123 Feb 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed


// Allocate new bag outside classTrackLock lock to avoid deadlock
struct bag* new_bag = bagCreateBag(sizeof(char*), 10);
debugMonitorEnter(classTrackLock);
if (deletedSignatures == NULL) {
deletedSignatures = new_bag;
new_bag = NULL;
}
debugMonitorExit(classTrackLock);
bagDestroyBag(new_bag);
Copy link
Contributor

@plummercj plummercj Feb 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One other comment about classTrack_processUnloads() in general that also applies to the original version. It seems pretty inefficient. Every time the debug agent gets an event it is called. On almost ever call it is returning and empty back and allocating a new one. It seems a check for bagSize(deletedSignatures) == 0 and returning NULL if true would help performance. I also believe this can be done outside of the lock (would like David's opinion on this).

Copy link
Contributor Author

@zhengyu123 zhengyu123 Feb 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure if it is possible, but checking bagSize(deletedSignatures) == 0 seems to race against classTrack_reset() where it does not take handlerLock lock.

Copy link
Contributor

@plummercj plummercj Feb 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure if it is possible, but checking bagSize(deletedSignatures) == 0 seems to race against classTrack_reset() where it does not take handlerLock lock.

I had thought of that too, but I think the way classTrack_reset() is called, it is likely not possible for there to be a classTrack_processUnloads() also coming in because everything is shut down:

    threadControl_onDisconnect();
    standardHandlers_onDisconnect();

    /*
     * Cut off the transport immediately. This has the effect of
     * cutting off any events that the eventHelper thread might
     * be trying to send.
     */
    transport_close();
    debugMonitorDestroy(cmdQueueLock);

    /* Reset for a new connection to this VM if it's still alive */
    if ( ! gdata->vmDead ) {
        debugInit_reset(getEnv());    <--- calls classTrack_reset()
    }

Copy link
Contributor Author

@zhengyu123 zhengyu123 Feb 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But shutting down debug loop does not seem to have effect on ongoing jvmti callback, e.g. thread 4 in the bug report.

Copy link
Contributor

@plummercj plummercj Feb 16, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The thread 4 callback is triggered due to the debugger having requested CLASS_UNLOAD events. This is shutdown by eventHandler_reset(), which is called by debugInit_reset() before it calls classTrack_reset(). So that means by the time we get to classTrack_reset(), the thread 4 callback is no longer possible.

return deleted;
}

@@ -194,8 +212,9 @@ classTrack_initialize(JNIEnv *env)
void
classTrack_activate(JNIEnv *env)
{
struct bag* new_bag = bagCreateBag(sizeof(char*), 1000);
Copy link
Member

@dholmes-ora dholmes-ora Feb 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think there can be any race during activation but this change is harmless.

Copy link
Contributor Author

@zhengyu123 zhengyu123 Feb 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure classTrack_activate() has the problem, but it is called outside of initialization, inside debug loop ... so I assume that event callback might happen.

Copy link
Contributor

@plummercj plummercj Feb 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree. This code is called when the debug agent receives a CLASS_UNLOAD EventRequest (which the debug agent handles by requesting JVMTI GC_FINISH events). So there is nothing that prevents classTrack_processUnloads() from being called at the same time.

Copy link
Member

@dholmes-ora dholmes-ora Feb 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In that case the classTrack_processUnloads is racing with the classTrack_activate call. I would not expect it to be possible to generate an event in relation to this before things have been "activated". But moving on ...

Copy link
Member

@dholmes-ora dholmes-ora Feb 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay I see from later comments that in fact no such race exists, but we have to synchronize with cbTrackingObjectFree

Copy link
Member

@dholmes-ora dholmes-ora Feb 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bah! But then a comment after that says it does exist again.

There seem to be issues understanding exactly what the concurrent call sequences can be with this code.

Copy link
Contributor

@plummercj plummercj Feb 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you might be confusing classTrack_activate() with classTrack_reset(). I believe the former has a race but the latter does not.

Copy link
Contributor

@plummercj plummercj Feb 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

...and even with the race, I still I think it is safe for a new bagSize(deletedSignatures) == 0 check to be added outside the lock. The only thing classTrack_activate() can do to deletedSignatures is set it when it is null. There is no risk of deletedSignatures being deleted and reallocated by another thread once you are in classTrack_processUnloads(). This is because it is called while holding the handlerLock, and as I explained elsewhere, by the time classTrack_reset() is called, it's no longer possible to be in classTrack_processUnloads().

debugMonitorEnter(classTrackLock);
deletedSignatures = bagCreateBag(sizeof(char*), 1000);
deletedSignatures = new_bag;
debugMonitorExit(classTrackLock);
}

@@ -213,13 +232,15 @@ cleanDeleted(void *signatureVoid, void *arg)
void
classTrack_reset(void)
{
struct bag* to_delete = NULL;
debugMonitorEnter(classTrackLock);

if (deletedSignatures != NULL) {
bagEnumerateOver(deletedSignatures, cleanDeleted, NULL);
bagDestroyBag(deletedSignatures);
to_delete = deletedSignatures;
Copy link
Member

@dholmes-ora dholmes-ora Feb 14, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cleanDeleted also calls jvmtiDeallocate, so either both need to be outside the lock or neither.

It is really unclear to me how many threads can be involved here and which functions can be called when.

Copy link
Contributor Author

@zhengyu123 zhengyu123 Feb 15, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right. Moved bagEnumerateOver() outside of lock as well.

deletedSignatures = NULL;
}

debugMonitorExit(classTrackLock);
bagDestroyBag(to_delete);
}