Support Claude 3

[Osiris-Team]

Recently claude.ai released their latest model that seems to be much better than ChatGPT, see: https://www.anthropic.com/news/claude-3-family

Thus I was wondering if support for it can be added in this plugin, or if there is already support for it, how to set that up?

[phodal]

You can try use https://ide.unitmesh.cc/custom/llm-server to customize your LLM.

[Osiris-Team]

@phodal asked claude, does this seem reasonable?

To integrate Claude 3 into the IntelliJ IDE plugin that supports multiple LLMs, you'll need to configure it as a custom LLM server. Based on the documentation you've provided, here's how you can add Claude 3:

1. Custom Response Type:
   Claude 3 typically uses Server-Sent Events (SSE) for streaming responses, so you should set the Custom Response Type to SSE.

2. Custom Engine Server:
   Use the Anthropic API endpoint for Claude. The exact URL may vary depending on the specific Claude 3 model you're using, but it's typically something like:

https://api.anthropic.com/v1/messages

3. Request body format:
   You'll need to customize the request body to match Anthropic's API requirements. Here's an example of how you might set it up:

{
  "customFields": {
    "model": "claude-3-opus-20240229",
    "stream": true
  },
  "messageKeys": {
    "role": "role",
    "content": "content"
  },
  "customHeaders": {
    "anthropic-version": "2023-06-01",
    "x-api-key": "YOUR_API_KEY_HERE"
  }
}

Note: Replace "claude-3-opus-20240229" with the specific Claude 3 model you're using (e.g., "claude-3-sonnet-20240229" for Claude 3 Sonnet).

4. Response format:
   Claude's streaming response format is similar to OpenAI's, so you can use the following JSON path:

$.delta.text

This will extract the content from each chunk of the streamed response.

5. Additional considerations:

• Make sure to include your Anthropic API key in the x-api-key header.
• The anthropic-version header is required by Anthropic's API.
• You may need to adjust the message structure slightly, as Anthropic's API expects messages in a specific format.

6. Implementing the integration:
   When implementing this in your plugin, you'll need to ensure that:

• The API key is securely stored and retrieved.
• The messages are properly formatted according to Anthropic's API requirements.
• Any Claude-specific parameters (like max_tokens, temperature, etc.) are properly included in the request if needed.

Remember to test the integration thoroughly, as there might be slight differences in how Claude 3 handles certain requests compared to other LLMs.

Would you like me to elaborate on any part of this integration process?

[phodal]

The AI's response make sense, but Claude cannot use in my country, I don't have API key to test it.

[Osiris-Team]

@phodal this seems to fail, here is my config:

Maybe it needs some changes to the code for this to work. I also tried it first with providing the token/secret-key in the json request, this instead resulted in a error 400, then without it it resulted in error 401:

2024-07-02 13:18:52,554 [4484484] SEVERE - #cc.unitmesh.devti.llms.custom.ResponseBodyCallback - IntelliJ IDEA 2024.1.3  Build #IC-241.17890.1
2024-07-02 13:18:52,554 [4484484] SEVERE - #cc.unitmesh.devti.llms.custom.ResponseBodyCallback - JDK: 17.0.11; VM: OpenJDK 64-Bit Server VM; Vendor: JetBrains s.r.o.
2024-07-02 13:18:52,555 [4484485] SEVERE - #cc.unitmesh.devti.llms.custom.ResponseBodyCallback - OS: Windows 11
2024-07-02 13:18:52,555 [4484485] SEVERE - #cc.unitmesh.devti.llms.custom.ResponseBodyCallback - Plugin to blame: AutoDev version: 1.8.7-RELEASE-241
2024-07-02 13:18:52,555 [4484485] SEVERE - #cc.unitmesh.devti.llms.custom.ResponseBodyCallback - Last Action: EditorPaste
2024-07-02 13:18:52,556 [4484486]   INFO - STDERR - java.io.IOException: AutoDevHttpException(statusCode=400, message=okhttp3.internal.http.RealResponseBody@e94bd0e)
2024-07-02 13:18:52,556 [4484486]   INFO - STDERR - 	at cc.unitmesh.devti.llms.custom.ResponseBodyCallback.onResponse(ResponseBodyCallback.kt:124)
2024-07-02 13:18:52,556 [4484486]   INFO - STDERR - 	at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:504)
2024-07-02 13:18:52,556 [4484486]   INFO - STDERR - 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
2024-07-02 13:18:52,556 [4484486]   INFO - STDERR - 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
2024-07-02 13:18:52,556 [4484486]   INFO - STDERR - 	at java.base/java.lang.Thread.run(Thread.java:840)
2024-07-02 13:18:52,557 [4484487]   INFO - STDERR - Caused by: AutoDevHttpException(statusCode=400, message=okhttp3.internal.http.RealResponseBody@e94bd0e)
2024-07-02 13:18:52,557 [4484487]   INFO - STDERR - 	at cc.unitmesh.devti.llms.custom.ResponseBodyCallback.onResponse(ResponseBodyCallback.kt:58)
2024-07-02 13:18:52,557 [4484487]   INFO - STDERR - 	... 4 more
2024-07-02 13:20:02,679 [4554609]   WARN - #c.i.o.o.e.ConfigurableCardPanel - auto-dispose 'Plugins' id=preferences.pluginManager
2024-07-02 13:20:02,697 [4554627]   INFO - STDOUT - false
2024-07-02 13:20:02,977 [4554907]   INFO - #c.i.w.i.i.j.s.JpsGlobalModelSynchronizerImpl - Saving global entities com.intellij.platform.workspace.jps.entities.SdkEntity to files
2024-07-02 13:20:02,977 [4554907]   INFO - #c.i.w.i.i.j.s.JpsGlobalModelSynchronizerImpl - Saving global entities com.intellij.platform.workspace.jps.entities.LibraryEntity to files
2024-07-02 13:20:02,988 [4554918]   INFO - #c.i.a.o.PathMacrosImpl - Saved path macros: {MAVEN_REPOSITORY=C:\Users\arman\.m2\repository}
2024-07-02 13:20:02,993 [4554923]   WARN - #c.i.u.x.Binding - No accessors for java.time.ZonedDateTime. This means that state class cannot be serialized properly. Please see https://jb.gg/ij-psoc
2024-07-02 13:20:05,903 [4557833]   WARN - #c.i.o.o.e.ConfigurableExtensionPointUtil - ignore deprecated groupId: language for id: preferences.language.Kotlin.scripting
2024-07-02 13:20:05,958 [4557888]   INFO - STDOUT - false
2024-07-02 13:20:07,197 [4559127]   INFO - #cc.unitmesh.devti.llms.custom.CustomLLMProvider - Requesting form: {"messages":[{"role":"user","content":"hi"}],"model":"claude-3-opus-20240229","stream":true} okhttp3.RequestBody$Companion$toRequestBody$2@31faabfe
2024-07-02 13:20:07,370 [4559300] SEVERE - #cc.unitmesh.devti.llms.custom.ResponseBodyCallback - Error while reading SSE
AutoDevHttpException(statusCode=401, message=okhttp3.internal.http.RealResponseBody@42e15add)
	at cc.unitmesh.devti.llms.custom.ResponseBodyCallback.onResponse(ResponseBodyCallback.kt:58)
	at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:504)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
2024-07-02 13:20:07,371 [4559301] SEVERE - #cc.unitmesh.devti.llms.custom.ResponseBodyCallback - IntelliJ IDEA 2024.1.3  Build #IC-241.17890.1
2024-07-02 13:20:07,371 [4559301] SEVERE - #cc.unitmesh.devti.llms.custom.ResponseBodyCallback - JDK: 17.0.11; VM: OpenJDK 64-Bit Server VM; Vendor: JetBrains s.r.o.
2024-07-02 13:20:07,371 [4559301] SEVERE - #cc.unitmesh.devti.llms.custom.ResponseBodyCallback - OS: Windows 11
2024-07-02 13:20:07,371 [4559301] SEVERE - #cc.unitmesh.devti.llms.custom.ResponseBodyCallback - Plugin to blame: AutoDev version: 1.8.7-RELEASE-241
2024-07-02 13:20:07,371 [4559301] SEVERE - #cc.unitmesh.devti.llms.custom.ResponseBodyCallback - Last Action: ShowSettings
2024-07-02 13:20:07,371 [4559301]   INFO - STDERR - java.io.IOException: AutoDevHttpException(statusCode=401, message=okhttp3.internal.http.RealResponseBody@42e15add)
2024-07-02 13:20:07,372 [4559302]   INFO - STDERR - 	at cc.unitmesh.devti.llms.custom.ResponseBodyCallback.onResponse(ResponseBodyCallback.kt:124)
2024-07-02 13:20:07,372 [4559302]   INFO - STDERR - 	at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:504)
2024-07-02 13:20:07,372 [4559302]   INFO - STDERR - 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
2024-07-02 13:20:07,372 [4559302]   INFO - STDERR - 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
2024-07-02 13:20:07,372 [4559302]   INFO - STDERR - 	at java.base/java.lang.Thread.run(Thread.java:840)
2024-07-02 13:20:07,372 [4559302]   INFO - STDERR - Caused by: AutoDevHttpException(statusCode=401, message=okhttp3.internal.http.RealResponseBody@42e15add)
2024-07-02 13:20:07,372 [4559302]   INFO - STDERR - 	at cc.unitmesh.devti.llms.custom.ResponseBodyCallback.onResponse(ResponseBodyCallback.kt:58)
2024-07-02 13:20:07,372 [4559302]   INFO - STDERR - 	... 4 more

Here some more info about the claude api: https://docs.anthropic.com/en/api/getting-started

[phodal]

Please follow the official doc to setup: https://docs.anthropic.com/en/api/getting-started

the key should be in head: x-api-key

[Osiris-Team]

@phodal yeah the question is how to do this in auto-dev?
Ah was it with custom-headers in json? Yeah I tried that, error 400 as I said, see error log above.

[phodal]

@iptton any idea ?

[Osiris-Team]

Going through their docs, and error 400 means

400 - invalid_request_error: There was an issue with the format or content of your request. We may also use this error type for other 4XX status codes not listed below.

at least its not error 401 anymore, so I guess authentication works by providing the key in the custom-headers json request.
The error response also has a json body, it would be great to see that in the log output too, maybe it contains more details about what the specific error is.

[Osiris-Team]

They got a python client available, maybe looking at that source code could also help out to integrate this into auto-dev:
https://github.com/anthropics/anthropic-sdk-python

[phodal]

@Osiris-Team I try the same thing in Insomnia/Postman and AutoDev, it return same error.

{
	"type": "error",
	"error": {
		"type": "invalid_request_error",
		"message": "Your credit balance is too low to access the Claude API. Please go to Plans & Billing to upgrade or purchase credits."
	}
}

Since I don't have a mobile phone out China, I update the response code in AutoDev. So you can see the error message now:

[phodal]

You can download the new version, after this action finish: https://github.com/unit-mesh/auto-dev/actions/runs/9762000493

[Osiris-Team]

@phodal thanks! I guess that's really it, I assumed they would let you use the API for free since their frontend can be used for free, but it seems that each API request must be paid.