IntelliJ Crashing - potentially due to variance in output formatting

[singerbj]

Before submitting your bug report

• I believe this is a bug. I'll try to join the Continue Discord for questions
• I'm not able to find an open issue that reports the same bug
• I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

- OS: Windows 11
- Continue: 0.0.64
- IDE: IntelliJ 2023.2.4
- Model: meta-llama-Meta-Llama-3.1-70B-Instruct-awq
- config.json:
  
{
  "models": [
    {
      "title": "Chat",
      "model": "meta-llama-Meta-Llama-3.1-70B-Instruct-awq",
      "provider": "openai",
      "apiKey": "<API_KEY>",
      "apiBase": "<API_BASE>",
      "contextLength": 16000,
      "maxTokens": 4096,
      "systemMessage": "Don't hallucinate."
    }
  ],
  "tabAutocompleteModel": {
    "title": "Tab Autocomplete Model",
    "provider": "openai",
    "model": "deepseek-ai-deepseek-coder-6.7b-base-awq",
    "apiKey": "<API_KEY>",
    "apiBase": "<API_BASE>",
    "completionOptions": {
      "presencePenalty": 1.1,
      "frequencyPenalty": 1.1
    }
  },
  "allowAnonymousTelemetry": false
}

Description

At a certain point, the output of the model in use with Continue starts looking like this in IntelliJ:

This could just be hallucination from our model...but ultimately through debugging, it seems that the variance in fonts shown above causes the editor to crash.

Heapdump: https://file.io/TWQ3wnTrnehZ

A place to look is between:

AWT-EventQueue-0
and
DefaultDispatcher-worker-14

It seems he plugin is flooding the eventqueue with events. There are 1700 in there waiting to be processed, but I think something is spinning them out and keeping the queue flooded, that's why my IDE seizes up.

Here are some relevant stacktraces:

It looks like it is hanging while trying to dispose a window:

DefaultDispatcher-worker-14
  at java.lang.Object.wait(J)V (Object.java(Native Method))
  at java.lang.Object.wait()V (Object.java:338)
  at java.awt.EventQueue.invokeAndWait(Ljava/lang/Object;Ljava/lang/Runnable;)V (EventQueue.java:1387)
  at java.awt.Window.doDispose()V (Window.java:1254)
  at java.awt.Window.dispose()V (Window.java:1191)
  at com.intellij.ui.jcef.HwFacadeHelper.removeNotify()V (HwFacadeHelper.java:230)
  at com.intellij.ui.jcef.HwFacadeJPanel.removeNotify()V (HwFacadeJPanel.java:25)
  at com.intellij.ui.BalloonImpl$MyComponent.removeNotify()V (BalloonImpl.java:1988)
  at java.awt.Container.remove(I)V (Container.java:1235)
  at javax.swing.JLayeredPane.remove(I)V (JLayeredPane.java:248)
  at java.awt.Container.lambda$remove$1(Ljava/awt/Component;)V (Container.java:1293)
  at java.awt.Container$$Lambda$888+0x00000008009c42b8.run()V ()
  at sun.awt.SunToolkit.lambda$performWithTreeLock$1(Ljava/lang/Runnable;)V (SunToolkit.java:2139)
  at sun.awt.SunToolkit$$Lambda$400+0x0000000800377a80.run()V ()
  at sun.awt.SunToolkit.performOnMainThreadIfNeeded(Ljava/lang/Runnable;)V (SunToolkit.java:2164)
  at sun.awt.SunToolkit.performWithTreeLock(Ljava/lang/Runnable;)V (SunToolkit.java:2137)
  at java.awt.Container.remove(Ljava/awt/Component;)V (Container.java:1289)
  at com.intellij.ui.BalloonImpl.show(Lcom/intellij/util/ui/PositionTracker;Lcom/intellij/ui/BalloonImpl$AbstractPosition;)V (BalloonImpl.java:566)
  at com.intellij.ui.BalloonImpl.show(Lcom/intellij/ui/awt/RelativePoint;Lcom/intellij/ui/BalloonImpl$AbstractPosition;)V (BalloonImpl.java:450)
  at com.intellij.ui.BalloonImpl.show(Lcom/intellij/ui/awt/RelativePoint;Lcom/intellij/openapi/ui/popup/Balloon$Position;)V (BalloonImpl.java:421)
  at com.github.continuedev.continueintellijextension.continue.IdeProtocolClient.showMessage(Ljava/lang/String;)V (IdeProtocolClient.kt:971)
  at com.github.continuedev.continueintellijextension.continue.IdeProtocolClient$handleMessage$1.invokeSuspend(Ljava/lang/Object;)Ljava/lang/Object; (IdeProtocolClient.kt:586)
  at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(Ljava/lang/Object;)V (ContinuationImpl.kt:33)
  at kotlinx.coroutines.DispatchedTask.run()V (DispatchedTask.kt:106)
  at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run()V (LimitedDispatcher.kt:115)
  at kotlinx.coroutines.scheduling.TaskImpl.run()V (Tasks.kt:100)
  at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(Lkotlinx/coroutines/scheduling/Task;)V (CoroutineScheduler.kt:584)
  at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(Lkotlinx/coroutines/scheduling/Task;)V (CoroutineScheduler.kt:793)
  at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker()V (CoroutineScheduler.kt:697)
  at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run()V (CoroutineScheduler.kt:684)

This one makes it look like there is a live lock happening with the chat plugin. It appears the overuse of changing fonts is the cause of this.

AWT-EventQueue-0
  at java.awt.Component.setFont(Ljava/awt/Font;)V (Component.java:1966)
  at java.awt.Container.setFont(Ljava/awt/Font;)V (Container.java:1780)
  at javax.swing.JComponent.setFont(Ljava/awt/Font;)V (JComponent.java:2810)
  at javax.swing.LookAndFeel.installColorsAndFont(Ljavax/swing/JComponent;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)V (LookAndFeel.java:213)
  at javax.swing.plaf.basic.BasicPanelUI.installDefaults(Ljavax/swing/JPanel;)V (BasicPanelUI.java:82)
  at javax.swing.plaf.basic.BasicPanelUI.installUI(Ljavax/swing/JComponent;)V (BasicPanelUI.java:67)
  at javax.swing.JComponent.setUI(Ljavax/swing/plaf/ComponentUI;)V (JComponent.java:733)
  at javax.swing.JPanel.setUI(Ljavax/swing/plaf/PanelUI;)V (JPanel.java:153)
  at javax.swing.JPanel.updateUI()V (JPanel.java:129)
  at javax.swing.JPanel.<init>(Ljava/awt/LayoutManager;Z)V (JPanel.java:89)
  at javax.swing.JPanel.<init>(Z)V (JPanel.java:112)
  at javax.swing.JPanel.<init>()V (JPanel.java:120)
  at com.intellij.ui.jcef.HwFacadeHelper$4.<init>(Lcom/intellij/ui/jcef/HwFacadeHelper;)V (HwFacadeHelper.java:188)
  at com.intellij.ui.jcef.HwFacadeHelper.activateIfNeeded(Ljava/util/List;)V (HwFacadeHelper.java:188)
  at com.intellij.ui.jcef.HwFacadeHelper$2.componentResized(Ljava/awt/event/ComponentEvent;)V (HwFacadeHelper.java:142)
  at java.awt.Component.processComponentEvent(Ljava/awt/event/ComponentEvent;)V (Component.java:6483)
  at java.awt.Component.processEvent(Ljava/awt/AWTEvent;)V (Component.java:6437)
  at java.awt.Container.processEvent(Ljava/awt/AWTEvent;)V (Container.java:2266)
  at java.awt.Component.dispatchEventImpl(Ljava/awt/AWTEvent;)V (Component.java:5027)
  at java.awt.Container.dispatchEventImpl(Ljava/awt/AWTEvent;)V (Container.java:2324)
  at java.awt.Component.dispatchEvent(Ljava/awt/AWTEvent;)V (Component.java:4855)
  at java.awt.EventQueue.dispatchEventImpl(Ljava/awt/AWTEvent;Ljava/lang/Object;)V (EventQueue.java:794)
  at java.awt.EventQueue$3.run()Ljava/lang/Void; (EventQueue.java:739)
  at java.awt.EventQueue$3.run()Ljava/lang/Object; (EventQueue.java:733)
  at java.security.AccessController.executePrivileged(Ljava/security/PrivilegedAction;Ljava/security/AccessControlContext;Ljava/lang/Class;)Ljava/lang/Object; (AccessController.java:776)
  at java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;Ljava/security/AccessControlContext;)Ljava/lang/Object; (AccessController.java:399)
  at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Ljava/security/PrivilegedAction;Ljava/security/AccessControlContext;Ljava/security/AccessControlContext;)Ljava/lang/Object; (ProtectionDomain.java:86)
  at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Ljava/security/PrivilegedAction;Ljava/security/AccessControlContext;)Ljava/lang/Object; (ProtectionDomain.java:97)
  at java.awt.EventQueue$4.run()Ljava/lang/Void; (EventQueue.java:766)
  at java.awt.EventQueue$4.run()Ljava/lang/Object; (EventQueue.java:764)
  at java.security.AccessController.executePrivileged(Ljava/security/PrivilegedAction;Ljava/security/AccessControlContext;Ljava/lang/Class;)Ljava/lang/Object; (AccessController.java:776)
  at java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;Ljava/security/AccessControlContext;)Ljava/lang/Object; (AccessController.java:399)
  at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Ljava/security/PrivilegedAction;Ljava/security/AccessControlContext;Ljava/security/AccessControlContext;)Ljava/lang/Object; (ProtectionDomain.java:86)
  at java.awt.EventQueue.dispatchEvent(Ljava/awt/AWTEvent;)V (EventQueue.java:763)
  at com.intellij.ide.IdeEventQueue.defaultDispatchEvent(Ljava/awt/AWTEvent;)V (IdeEventQueue.kt:690)
  at com.intellij.ide.IdeEventQueue._dispatchEvent$lambda$10(Lcom/intellij/ide/IdeEventQueue;Ljava/awt/AWTEvent;)V (IdeEventQueue.kt:593)
  at com.intellij.ide.IdeEventQueue$$Lambda$512+0x00000008005d0690.run()V ()
  at com.intellij.openapi.application.impl.ApplicationImpl.runWithoutImplicitRead(Ljava/lang/Runnable;)V (ApplicationImpl.java:1485)
  at com.intellij.ide.IdeEventQueue._dispatchEvent(Ljava/awt/AWTEvent;)V (IdeEventQueue.kt:593)
  at com.intellij.ide.IdeEventQueue.access$_dispatchEvent(Lcom/intellij/ide/IdeEventQueue;Ljava/awt/AWTEvent;)V (IdeEventQueue.kt:67)
  at com.intellij.ide.IdeEventQueue$dispatchEvent$processEventRunnable$1$1$1.compute()Ljava/lang/Void; (IdeEventQueue.kt:369)
  at com.intellij.ide.IdeEventQueue$dispatchEvent$processEventRunnable$1$1$1.compute()Ljava/lang/Object; (IdeEventQueue.kt:368)
  at com.intellij.openapi.progress.impl.CoreProgressManager.computePrioritized(Lcom/intellij/openapi/util/ThrowableComputable;)Ljava/lang/Object; (CoreProgressManager.java:787)
  at com.intellij.ide.IdeEventQueue$dispatchEvent$processEventRunnable$1$1.invoke()V (IdeEventQueue.kt:368)
  at com.intellij.ide.IdeEventQueue$dispatchEvent$processEventRunnable$1$1.invoke()Ljava/lang/Object; (IdeEventQueue.kt:363)
  at com.intellij.ide.IdeEventQueueKt.performActivity$lambda$1(Lkotlin/jvm/functions/Function0;)V (IdeEventQueue.kt:997)
  at com.intellij.ide.IdeEventQueueKt$$Lambda$505+0x00000008005c87d0.run()V ()
  at com.intellij.openapi.application.TransactionGuardImpl.performActivity(ZLjava/lang/Runnable;)V (TransactionGuardImpl.java:105)
  at com.intellij.ide.IdeEventQueueKt.performActivity(Ljava/awt/AWTEvent;Lkotlin/jvm/functions/Function0;)V (IdeEventQueue.kt:997)
  at com.intellij.ide.IdeEventQueue.dispatchEvent$lambda$7(Lcom/intellij/ide/IdeEventQueue;Ljava/awt/AWTEvent;ZLjava/awt/AWTEvent;Lcom/intellij/diagnostic/EventWatcher;Ljava/lang/Runnable;Ljava/lang/Class;J)V (IdeEventQueue.kt:363)
  at com.intellij.ide.IdeEventQueue$$Lambda$501+0x00000008005c4268.run()V ()
  at com.intellij.openapi.application.impl.ApplicationImpl.runIntendedWriteActionOnCurrentThread(Ljava/lang/Runnable;)V (ApplicationImpl.java:861)
  at com.intellij.ide.IdeEventQueue.dispatchEvent(Ljava/awt/AWTEvent;)V (IdeEventQueue.kt:405)
  at java.awt.EventDispatchThread.pumpOneEventForFilters(I)V (EventDispatchThread.java:207)
  at java.awt.EventDispatchThread.pumpEventsForFilter(ILjava/awt/Conditional;Ljava/awt/EventFilter;)V (EventDispatchThread.java:128)
  at java.awt.EventDispatchThread.pumpEventsForHierarchy(ILjava/awt/Conditional;Ljava/awt/Component;)V (EventDispatchThread.java:117)
  at java.awt.EventDispatchThread.pumpEvents(ILjava/awt/Conditional;)V (EventDispatchThread.java:113)
  at java.awt.EventDispatchThread.pumpEvents(Ljava/awt/Conditional;)V (EventDispatchThread.java:105)
  at java.awt.EventDispatchThread.run()V (EventDispatchThread.java:92)

To reproduce

We think....

1. Use a model that starts to hallucinate, causing rapid variance in text formatting (like pictured above)
2. Have enough variance that the plugin crashes :(

Log output

core.log

[sestinj]

It seems like there are two separate issues here, as I'm quite confident that the hallucinations are not causing the crash. The reason I'm so confident is that (assuming the font changes are to blame) the Continue sidebar does not use javax.swing (it is rendered as HTML).

Given this, I think it's best to first try and untangle the hallucination, and I do have a possible explanation for this. If a message was sent, the stream was cancelled, and then a new message was sent just after I've noticed (on very rare occasion) that there's a bug that will cause it to interleave the two streams, which makes for a very jumbled response.

Is the model typically giving sensibly formatted responses except in this one case? If so, then I think that may answer this part of the puzzle

[blinkus2000]

I found the live-lock with the font rendering/event spam and the disposal of the window between the two threads.

It's in the provided heap dump. I did not know the source of the event spam. I just guessed since both things were happening at the same time. I did not spend too much time in the heap, but those two threads immediately stood out.

The formatting/hallucination is quite easy to reproduce, if the prompt leans more towards conversational than specific and concise.

Honestly it is easy enough to work around. The bigger issue is the plugin locking up my IDE requiring frequent restarts.

[singerbj]

@blinkus2000 had me upload another zip: https://file.io/rkLsUi5bbDBj

A heap dump while the IDE is hung.

I did very minor digging into the heap and isolated the threads touching anything continue.dev for their convenience.
I included 3 text files.
threadsTouchingContinueDev.txt
stacks.txt //the stacks of the threads.
and the OQL query to pull them up themselves and take a peek.

"It is prety bad, I have to manually kill the process in the task manager."

[blinkus2000]

OK, I think I solved the issue. There are UI operations happening in the following block:

coroutineScope.launch(Dispatchers.IO) {}

When dealing with the UI doing things in the Dispatchers.IO can deadlock, you need to do UI work in Dispatchers.Main iirc. Not an expert in kotlin but the deadlock in the heap is very real.

[sestinj]

Thanks y'all for looking into this. I've made the necessary updates as far as I can tell, and also just did a full pass on the codebase to clean up whatever potential concurrency flaws I could find. This is currently available in EAP 0.0.69, and will be in a main release very soon (just waiting on the JetBrains review process)

[chubshaun]

Is there anything we can provide (specific logs, dumps) to help narrow this down? This is the second time Jetbrains Webstorm has completely soft-locked (beach ball) requiring me to force kill the entire process on my Mac M1 and I've been using the latest version of Continue dev extension 0.0.69.

For reference, I tend to run multiple instances of Webstorm simultaneously but against different repo folders.

[blinkus2000]

I was just using JConsole to capture a heapdump of Intellij then analyzing it in MAT to help them narrow the problem down.

Once it is open in a histogram you can group the objects by package, find com.github.continuedev.continueintellijextension, right click and select merge on gc roots (ignore phantom, weak, etc)that will show you all the threads touching stuff from continue.dev from there select them all, right click -> java basics -> thread overview and stacks

That isolates the threads involved in the deadlock if it is still there. (this technique works for lots of other codebases too)

[sestinj]

I've not seen any related problems since making the update, can y'all confirm that this seems to be solved, or have there still been crashes?

[blinkus2000]

Yeah this is cleared up.

[github-actions]

This issue hasn't been updated in 90 days and will be closed after an additional 10 days without activity. If it's still important, please leave a comment and share any new information that would help us address the issue.