feat(amazonq): add image context support #5846

zuoyaofu · 2025-06-21T00:34:07Z

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)

Description

Checklist

My code follows the code style of this project
I have added tests to cover my changes
A short description of the change has been added to the CHANGELOG if the change is customer-facing in the IDE.
I have added metrics for my changes (if required)

License

I confirm that my contribution is made under the terms of the Apache 2.0 license.

...tware/aws/toolkits/jetbrains/services/amazonq/lsp/model/aws/chat/showOpenFileDialogParams.kt

rli · 2025-06-24T00:23:59Z

...s-community/src/software/aws/toolkits/jetbrains/services/amazonq/webview/BrowserConnector.kt

@@ -485,6 +487,17 @@ class BrowserConnector(
                        )
                    }
            }
+            OPEN_FILE_DIALOG -> {
+                handleChat(AmazonQChatServer.showOpenFileDialog, node)


can you walk me through the message flow? does it actually need to transit through flare before requesting the file picker?

Flow:

chat-client will send openFileDialog request to flare

Flare will send to extension the params, including canSelectMany (multiple files), filters (what file extension types are allowed)

Extension will then handle file selection logic and send back the list of URIs.

Flare will then send to chat-client the file uri to display in chat.

- does it actually need to transit through flare before requesting the file picker?

In @yzhangok's design, yes. Flare is also telling client what type of files are supported in file picker as well(step 2)

...ns-community/src/software/aws/toolkits/jetbrains/services/amazonq/toolwindow/AmazonQPanel.kt

bryceitoc9 · 2025-06-24T17:26:37Z

.../jetbrains-community/src/software/aws/toolkits/jetbrains/services/amazonq/webview/Browser.kt

@@ -122,7 +125,7 @@ class Browser(parent: Disposable, private val webUri: URI, val project: Project)
            <script type="text/javascript" charset="UTF-8" src="$webUri" defer onload="init()"></script>

            <script type="text/javascript">
-
+                let qChat = undefined


What is this doing?

To use qChat in window.handleNativeDrop and window.handleNativeNotify

rli · 2025-06-24T17:23:18Z

...ns-community/src/software/aws/toolkits/jetbrains/services/amazonq/toolwindow/AmazonQPanel.kt

+
+
+
+                                            val json = Gson().toJson(validImages).replace("\\", "\\\\").replace("'", "\\'")


if it's json why do we need to escape this

Let me find a more elegant way to handle this

...ns-community/src/software/aws/toolkits/jetbrains/services/amazonq/toolwindow/AmazonQPanel.kt

rli · 2025-06-24T17:24:25Z

...ns-community/src/software/aws/toolkits/jetbrains/services/amazonq/toolwindow/AmazonQPanel.kt

+                            // Set DropTarget on the browser component and its children
+                            browserInstance.component()?.let { component ->
+                                component.dropTarget = dropTarget
+                                // Also try setting on parent if needed


rli · 2025-06-24T17:24:34Z

...ns-community/src/software/aws/toolkits/jetbrains/services/amazonq/toolwindow/AmazonQPanel.kt

+                                }
+                            }
+
+                            // Set DropTarget on the browser component and its children


how are the children involved

We want the whole chat window to be "drag & drop zone"

...ns-community/src/software/aws/toolkits/jetbrains/services/amazonq/toolwindow/AmazonQPanel.kt

rli · 2025-06-24T17:25:10Z

...ns-community/src/software/aws/toolkits/jetbrains/services/amazonq/toolwindow/AmazonQPanel.kt

+                                            val maxDimension = 8000
+
+                                            for (file in fileList) {
+                                                val fileObj = file as? java.io.File ?: continue


JCEF does not propagate OS-level dragenter, dragOver and drop into DOM. As an alternative, enabling the native drag in JCEF, and let the native handling the drop event, and update the UI through JS bridge.
The same logic of filtering the images are implemented on both language server and JetBrain

no why do we need to do an identity check here
the JDK docs state

To transfer a list of files to/from Java (and the underlying platform) a DataFlavor of this type/subtype and representation class of java.util.List is used. Each element of the list is required/guaranteed to be of type java.io.File.

...ns-community/src/software/aws/toolkits/jetbrains/services/amazonq/toolwindow/AmazonQPanel.kt

rli · 2025-06-24T17:33:21Z

...munity/src/software/aws/toolkits/jetbrains/services/amazonq/lsp/AmazonQLanguageClientImpl.kt

+                    FileChooserDescriptorFactory.createSingleFileNoJarsDescriptor()
+                }
+
+                val chosenFiles = FileChooser.chooseFiles(descriptor, project, null)


why aren't images from this flow subject to constraints?

Yes, they are now with the new commit.

what about the file size and dimensional limits?

...ns-community/src/software/aws/toolkits/jetbrains/services/amazonq/toolwindow/AmazonQPanel.kt

+                                            dtde.dropComplete(false)
+                                        }
+                                    } catch (e: Exception) {
+                                        LOG.error("Failed to handle file drop operation", e.message)


...ns-community/src/software/aws/toolkits/jetbrains/services/amazonq/toolwindow/AmazonQPanel.kt

+    companion object {
+        private val LOG = getLogger<AmazonQPanel>()
+
+        fun getInstance(project: Project) = project.service<AmazonQPanel>()


rli · 2025-07-01T17:54:45Z

...munity/src/software/aws/toolkits/jetbrains/services/amazonq/lsp/AmazonQLanguageClientImpl.kt

+                val descriptor = when {
+                    params.canSelectFolders && params.canSelectFiles -> {
+                        if (params.canSelectMany) {
+                            FileChooserDescriptorFactory.createSingleFileOrFolderDescriptor()


how is this multiselect

rli · 2025-07-01T17:55:25Z

...tware/aws/toolkits/jetbrains/services/amazonq/lsp/model/aws/chat/ShowOpenFileDialogParams.kt

+    val canSelectFiles: Boolean = false,
+    val canSelectFolders: Boolean = false,
+    val canSelectMany: Boolean = false,
+    val filters: Map<String, List<String>> = emptyMap(),
+    val title: String? = null,


dont need default values

rli · 2025-07-01T17:55:50Z

...ity/src/software/aws/toolkits/jetbrains/services/amazonq/lsp/model/ExtendedClientMetadata.kt

@@ -25,6 +25,7 @@ data class DeveloperProfiles(
    val developerProfiles: Boolean,
    val mcp: Boolean,
    val pinnedContextEnabled: Boolean,
+    val imageContextEnabled: Boolean = true,


Suggested change

val imageContextEnabled: Boolean = true,

val imageContextEnabled: Boolean,

rli · 2025-07-01T17:57:51Z

...ns-community/src/software/aws/toolkits/jetbrains/services/amazonq/toolwindow/AmazonQPanel.kt

@@ -3,6 +3,7 @@

 package software.aws.toolkits.jetbrains.services.amazonq.toolwindow

+import com.google.gson.Gson


we only use Gson in the context of lsp4j

rli · 2025-07-01T17:59:29Z

...ns-community/src/software/aws/toolkits/jetbrains/services/amazonq/toolwindow/AmazonQPanel.kt

+                                            val fileList = transferable.getTransferData(DataFlavor.javaFileListFlavor) as List<*>
+
+                                            val errorMessages = mutableListOf<String>()
+                                            val validImages = mutableListOf<java.io.File>()


Suggested change

val validImages = mutableListOf<java.io.File>()

val validImages = mutableListOf<File>()

rli · 2025-07-01T18:00:50Z

...ns-community/src/software/aws/toolkits/jetbrains/services/amazonq/toolwindow/AmazonQPanel.kt

+    companion object {
+        private val LOG = getLogger<AmazonQPanel>()
+
+        fun getInstance(project: Project) = project.service<AmazonQPanel>()


rli · 2025-07-01T18:01:28Z

...munity/src/software/aws/toolkits/jetbrains/services/amazonq/lsp/AmazonQLanguageClientImpl.kt

+                    FileChooserDescriptorFactory.createSingleFileNoJarsDescriptor()
+                }
+
+                val chosenFiles = FileChooser.chooseFiles(descriptor, project, null)


what about the file size and dimensional limits?

rli · 2025-07-01T18:03:31Z

...ns-community/src/software/aws/toolkits/jetbrains/services/amazonq/toolwindow/AmazonQPanel.kt

+                                            val maxDimension = 8000
+
+                                            for (file in fileList) {
+                                                val fileObj = file as? java.io.File ?: continue


no why do we need to do an identity check here
the JDK docs state

To transfer a list of files to/from Java (and the underlying platform) a DataFlavor of this type/subtype and representation class of java.util.List is used. Each element of the list is required/guaranteed to be of type java.io.File.

zuoyaofu added 2 commits June 20, 2025 16:40

feat(q): enable image context

92b30dc

feat(q): add image context support

79f86f6

zuoyaofu changed the title ~~Image~~ feat(q): add image context support Jun 21, 2025

zuoyaofu changed the title ~~feat(q): add image context support~~ feat(amazonq): add image context support Jun 21, 2025

zuoyaofu assigned zuoyaofu and yzhangok Jun 21, 2025

add change log + fix some styles

9d0fd61

rli reviewed Jun 24, 2025

View reviewed changes

pass the dropped files from Kotlin to webView

4621012

github-advanced-security bot found potential problems Jun 24, 2025

View reviewed changes

...ns-community/src/software/aws/toolkits/jetbrains/services/amazonq/toolwindow/AmazonQPanel.kt Fixed Show fixed Hide fixed

bryceitoc9 reviewed Jun 24, 2025

View reviewed changes

rli reviewed Jun 24, 2025

View reviewed changes

zuoyaofu and others added 4 commits June 25, 2025 14:03

use params to select different file descriptor

eb596bd

Merge branch 'main' into image

2594dbe

Merge branch 'main' into image

42016ee

add feature flag

3df8ef9

github-advanced-security bot found potential problems Jun 25, 2025

View reviewed changes

Merge branch 'main' into image

410c68e

zuoyaofu marked this pull request as ready for review June 26, 2025 22:38

zuoyaofu requested review from a team as code owners June 26, 2025 22:38

fixed comments

ac84d65

rli reviewed Jul 1, 2025

View reviewed changes




		val json = Gson().toJson(validImages).replace("\\", "\\\\").replace("'", "\\'")

	val imageContextEnabled: Boolean = true,
	val imageContextEnabled: Boolean,

		@@ -3,6 +3,7 @@

		package software.aws.toolkits.jetbrains.services.amazonq.toolwindow

		import com.google.gson.Gson

	val validImages = mutableListOf<java.io.File>()
	val validImages = mutableListOf<File>()

feat(amazonq): add image context support #5846

Are you sure you want to change the base?

feat(amazonq): add image context support #5846

Conversation

zuoyaofu commented Jun 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Types of changes

Description

Checklist

License

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zuoyaofu Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Check warning

Check failure

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

zuoyaofu commented Jun 21, 2025 •

edited

Loading

zuoyaofu Jun 25, 2025 •

edited

Loading