Invalid type of downloaded plugin

[lukaville]

Intellij plugin resolving fails with "Invalid type of downloaded plugin" error:

19:01:59.205 [INFO] [org.jetbrains.intellij.IntelliJPlugin] Downloading org.jetbrains.kotlin:1.1.1-release-IJ2016.3-1 from http://plugins.jetbrains.com
19:01:59.229 [INFO] [plugin-repository-rest-client] Downloading org.jetbrains.kotlin:1.1.1-release-IJ2016.3-1
19:01:59.237 [DEBUG] [plugin-repository-rest-client] ---> HTTP GET http://plugins.jetbrains.com/plugin/download?format=text&pluginId=org.jetbrains.kotlin&version=1.1.1-release-IJ2016.3-1
19:01:59.332 [DEBUG] [sun.net.www.protocol.http.HttpURLConnection] sun.net.www.MessageHeader@23c2bb195 pairs: {GET /plugin/download?format=text&pluginId=org.jetbrains.kotlin&version=1.1.1-release-IJ2016.3-1 HTTP/1.1: null}{User-Agent: Java/1.8.0_121}{Host: plugins.jetbrains.com}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2}{Connection: keep-alive}
19:01:59.402 [DEBUG] [sun.net.www.protocol.http.HttpURLConnection] sun.net.www.MessageHeader@7b372dd310 pairs: {null: HTTP/1.1 302}{Server: nginx}{Date: Thu, 18 May 2017 16:01:59 GMT}{Content-Length: 0}{Connection: keep-alive}{Set-Cookie: ccc=1ia4iu;Max-Age=157680000;Expires=Tue, 17-May-2022 16:01:59 GMT}{Location: http://plugins.jetbrains.com/files/6954/33474/kotlin-plugin-1.1.1-release-IJ2016.3-1.zip?updateId=33474&pluginId=6954}{X-Frame-Options: DENY}{X-Content-Type-Options: nosniff}{X-XSS-Protection: 1; mode=block;}
19:01:59.408 [DEBUG] [plugin-repository-rest-client] <--- HTTP 302 http://plugins.jetbrains.com/plugin/download?format=text&pluginId=org.jetbrains.kotlin&version=1.1.1-release-IJ2016.3-1 (170ms)
19:01:59.413 [DEBUG] [sun.net.www.protocol.http.HttpURLConnection] sun.net.www.MessageHeader@6a0768b65 pairs: {GET /files/6954/33474/kotlin-plugin-1.1.1-release-IJ2016.3-1.zip?updateId=33474&pluginId=6954 HTTP/1.1: null}{User-Agent: Java/1.8.0_121}{Host: plugins.jetbrains.com}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2}{Connection: keep-alive}
19:01:59.476 [DEBUG] [sun.net.www.protocol.http.HttpURLConnection] sun.net.www.MessageHeader@594a52ba10 pairs: {null: HTTP/1.1 302 Moved Temporarily}{Server: nginx}{Date: Thu, 18 May 2017 16:01:59 GMT}{Content-Type: text/html}{Content-Length: 154}{Connection: keep-alive}{Location: http://d9ernfkkvx1zk.cloudfront.net/6954/33474/kotlin-plugin-1.1.1-release-IJ2016.3-1.zip?updateId=33474&pluginId=6954}{X-Frame-Options: DENY}{X-Content-Type-Options: nosniff}{X-XSS-Protection: 1; mode=block;}
...
19:02:21.247 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] > Invalid type of downloaded plugin: /var/folders/hk/f9d556j567v_m_32vqqcyk9h29s5mm/T/intellij1489042748949457241/kotlin-plugin-1.1.1-release-IJ2016.3-1.zip?updateId=33474&pluginId=6954

It seems that something changed on jetbrains plugin repository backend. Now name of downloaded file contains GET params: kotlin-plugin-1.1.1-release-IJ2016.3-1.zip?updateId=33474&pluginId=6954 and zip/jar file type checks fail (https://github.com/JetBrains/gradle-intellij-plugin/blob/master/src/main/groovy/org/jetbrains/intellij/dependency/PluginDependencyManager.groovy#L125).

[baron1405]

Encountered the same problem this morning with the org.intellij.scala:2017.1.15 plugin so this appears to be systemic on the backend. As mentioned above, the downloaded filename now has the query string appended so the check in PluginDependencyManager for file type fails because the downloaded filename no longer ends in ".zip" or ".jar".

The plugin-repository-rest-client is hitting the following URL for the scala plugin mentioned above:

https://plugins.jetbrains.com/plugin/download?format=text&pluginId=org.intellij.scala&version=2017.1.15

As expected, the server is returning a 302 redirect. However, the location header must have changed, because it is now:

https://plugins.jetbrains.com/files/1347/33637/scala-intellij-bin-2017.1.15.zip?updateId=33637&pluginId=1347

and the downloadFile method is calling:

targetFile = File(targetFile, fileLocation.substringAfterLast('/'))

which explains why the filename on disk now has query parameters on it. So it looks like we are broken until plugin-repository-rest-client strips off the query parameters when it is forming the target filename.

[baron1405]

On a related note, while investigating this issue, I noticed that the http based URL for accessing the plugin repo specified by the gradle-intellij-plugin, causes the server to respond with a 307 redirect to the https based version of the URL. Looks like this is handled internally by some layer of the URL connection. However, the URL used by the plugin should be changed to https based to avoid the unnecessary redirect:

-public static final String DEFAULT_INTELLIJ_PLUGINS_REPO = 'http://plugins.jetbrains.com'
+public static final String DEFAULT_INTELLIJ_PLUGINS_REPO = 'https://plugins.jetbrains.com'

[lukaville]

It would be great if users can customize plugins repo url (and use custom binary artifacts storage, i.e. artifactory, and avoid such problems)

[zolotov]

@lukaville Gradle is flexible enough for this, just download the plugin from anywhere you want and attach it via intellij.plugins = [file('plugin.jar)]

[zolotov]

We're going to implement a workaround on the plugins repo side and deploy it tomorrow. The new version of gradle-intellij-plugin also will be available soon.

[lukaville]

@zolotov, I don't want to store binary files in version control repository :(
To make this thing work with external file storages I have to write a little bit hacky code with downloading plugin before dependency resolving.

[zolotov]

@zolotov, I don't want to store binary files in version control repository :(

Store it in artifactory as you suggested

I have to write a little bit hacky code with downloading plugin before dependency resolving.

I don't see any hacks in it and I don't understand why it should be done before dependency resolving. Actually, you can do this as a part of dependency resolving.

So, nothing from gradle-intellij-plugin is required for this.

[baron1405]

Thanks @zolotov for the quick response! Since I will have to use the new version of the plugin to pick up the fix, can you see if it possible to resolve #115 at the same time so I don't have to work around it (please)?

[zolotov]

@baron1405 I still not sure how to fix that issue, so I cannot promise, sorry. But I'll definitely take a look at it.

[lukaville]

@zolotov good point about implementing this as part of dependency resolving. Thanks, I'll try it.
Do you have any plans about migrating plugins repository to maven or to other popular repository type? So we can use repository proxies out of box, without writing code at all.

[zolotov]

@lukaville actually, I've already implemented a maven interface in plugins repository. It's quite hacky but it can be resolved by default Gradle resolver and cached by artifactory. Here is the description: http://www.jetbrains.org/intellij/sdk/docs/plugin_repository/api/maven_interface.html.

gradle-intellij-plugin cannot work with it yet but it's on the plan.

[zolotov]

Today we rolled back the fix at plugins.jetbrains.com, we're going to deploy it again tonight.

0.2.10 with the fix on the client's side is on the way. @baron1405 0.2.10 also contains adding external plugins to runtime classpath during the tests.

[baron1405]

Works great! Thanks @zolotov for the thoughtful discussion and quick resolution.