Support downloading the sources for attached felix bundles

[royteeuwen]

The attached felix bundles have a pom.xml most of the times. It would be very nice if you could download the sources of the attached bundle if they are available

[karollewandowski]

Hi Roy,
I love this idea and will try to implement it in the next release.
Thanks a lot for requesting this.

[karollewandowski]

Just a note about how easy it is to implement support for the AEM ecosystem 🙃

org.apache.sling.jcr.contentloader:2.2.6 - source JAR available in Maven Central, but it's missing one of packages 🙃
org.apache.sling.jcr.jcr-wrapper:2.0.0 - source JAR available in Maven Central, but it's missing all classes 🙃

Anyway, it will check whether the downloaded JAR matches local bundle classes and will propose decompilation in case they don't match.

[royteeuwen]

Haha tell me about it, it was fun developing the felix-bundle-extractor a few years ago ;)

[karollewandowski]

Hi, the initial implementation has been released to the EAP channel. Please update the plugin, test it, and provide feedback.

[royteeuwen]

Nice!

OK, so I tried to go into an open-source bundle and said "Fetch all bundles". The message I got was the following:

This should have been available though: https://mvnrepository.com/artifact/org.apache.felix/org.apache.felix.eventadmin/1.6.4

If I look at the actual jars, I see a lot of them is using sources, so not all are decompiled:

It also states that 77 could not be fetched, does that mean it decompiled those ones? Or what does that mean, the message might be clearer. And If I look at them, most of them are open source as well

Excerpt of the list:

[karollewandowski]

Hi Roy,
Thank you very much for testing it and providing feedback.

Regarding org.apache.felix.eventadmin:1.6.4, indeed, it should work. Maybe there was a connection issue, and the file couldn't be downloaded. If this is the case, the idea.log (Help | Open Log in Editor) should contain information about it. In the log file, search for co.nums.intellij.aem.BundleSources. Another possibility is that it doesn't contain META-INF/maven/.../pom.properties file, which we use for retrieving Maven coordinates (see the last paragraph for information what we do in such cases).

Regarding missing sources, it means that they couldn't be downloaded and decompiled (if decompilation was allowed). Would such a message work?

Could not download/decompile sources for 72 JARs. Do you want to display the missing sources list?

I checked com.cognifide.aem.stubs bundles, and their sources are not published to Maven Central. See:

• https://mvnrepository.com/artifact/com.cognifide.aem.stubs/core - version 3.0.2 is missing.
• https://mvnrepository.com/artifact/com.cognifide.aem.stubs/moco - version 3.0.2 is missing.

We can't do anything about it.

Not sure what is the reason for com.h2database.mvstore-2.1.214. Could you please check what is the META-INF/maven/.../pom.properties content in this bundle's JAR? In Maven Central it is h2-mvstore, not just mvstore. Maybe there is a discrepancy here? Or it is missing at all?

Unfortunately, the Fernflower decompiler can't handle some JARs. During my tests, it couldn't decompile, for example, JARs compiled from Scala sources (I guess Scala uses some non-standard bytecode instructions). Maybe this is also the case for Groovy. Unfortunately, I don't think we can do anything here. You can check in the log what the error is.

It's worth mentioning that the plugin contains a manual mapping of bundle symbolic names to Maven coordinates. This mapping was required because a few open-source bundles didn't have Maven coordinates. I tried to handle it with Solr search API at https://mvnrepository.com/, but unfortunately, it didn't work reliably, hence the manual mapping. Unfortunately, it's impossible to cover all bundles, as there are many AEM versions, and many custom bundles installed. I will add logging of bundles without Maven coordinates so that users can send them to me, and I will add them to the mapping (I think that the online mapping would be nice to have so that it could be changed without new releases).

[royteeuwen]

I checked my idea.log and didn't find any reference to co.nums.intellij.aem.BundleSources, only to co.nums.intellij.aem

I also see that the event.admin does have the pom:

Now that I try to re-open my intellij to check the things you asked, I don't see any of the felix bundles loaded anymore and in idea.log it states:

2024-11-13 21:06:13,718 [ 102098] SEVERE - #c.i.c.ComponentStoreImpl - Cannot init component state (componentName=OsgiSettings, componentClass=OsgiSettings) [Plugin: co.nums.intellij.aem]
com.intellij.diagnostic.PluginException: Cannot init component state (componentName=OsgiSettings, componentClass=OsgiSettings) [Plugin: co.nums.intellij.aem]
	at com.intellij.configurationStore.ComponentStoreImpl.initComponent(ComponentStoreImpl.kt:181)
	at com.intellij.configurationStore.ComponentStoreWithExtraComponents.initComponent(ComponentStoreWithExtraComponents.kt:44)
	at com.intellij.serviceContainer.ComponentManagerImpl.initializeService$intellij_platform_serviceContainer(ComponentManagerImpl.kt:659)
	at com.intellij.serviceContainer.LightServiceInstanceSupport$LightServiceInstanceInitializer.createInstance(LightServiceInstanceSupport.kt:56)
	at com.intellij.platform.instanceContainer.internal.LazyInstanceHolder$initialize$1$1.invokeSuspend(LazyInstanceHolder.kt:162)
	at com.intellij.platform.instanceContainer.internal.LazyInstanceHolder$initialize$1$1.invoke(LazyInstanceHolder.kt)
	at com.intellij.platform.instanceContainer.internal.LazyInstanceHolder$initialize$1$1.invoke(LazyInstanceHolder.kt)
	at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:78)
	at kotlinx.coroutines.BuildersKt__Builders_commonKt.withContext(Builders.common.kt:167)
Caused by: com.intellij.util.xmlb.XmlSerializationException: Cannot deserialize class co.nums.intellij.aem.osgi.felix.bundles.settings.OsgiSettings
	at com.intellij.configurationStore.StateStorageBaseKt.deserializeStateWithController(StateStorageBase.kt:369)
	at com.intellij.configurationStore.StateGetterImpl.getState(XmlElementStorage.kt:429)
	at com.intellij.configurationStore.ComponentStoreImpl.doInitComponent(ComponentStoreImpl.kt:569)
	at com.intellij.configurationStore.ComponentStoreImpl.initComponent(ComponentStoreImpl.kt:487)
	at com.intellij.configurationStore.ComponentStoreImpl.initComponent(ComponentStoreImpl.kt:156)
	... 103 more
Caused by: java.lang.NullPointerException: Cannot read the array length because "<local4>" is null
	at com.intellij.serialization.ClassUtil.stringToEnum(ClassUtil.java:103)
	at com.intellij.util.xmlb.XmlSerializerImpl.doSet(XmlSerializerImpl.java:215)
	at com.intellij.util.xmlb.TagBinding.deserialize(TagBinding.kt:243)
	at com.intellij.util.xmlb.BeanBindingKt.deserializeJdomIntoBean(BeanBinding.kt:425)
	at com.intellij.util.xmlb.BeanBinding.deserializeInto(BeanBinding.kt:198)
	at com.intellij.util.xmlb.BeanBinding.deserialize(BeanBinding.kt:188)
	at com.intellij.configurationStore.StateStorageBaseKt.deserializeStateWithController(StateStorageBase.kt:342)
	... 107 more

2024-11-13 21:07:28,445 [ 176825] SEVERE - #c.i.i.p.PluginManager - class co.nums.intellij.aem.osgi.felix.bundles.settings.OsgiSettings$a not an enum
java.lang.ClassCastException: class co.nums.intellij.aem.osgi.felix.bundles.settings.OsgiSettings$a not an enum
	at java.base/java.util.EnumSet.noneOf(EnumSet.java:113)
	at java.base/java.util.EnumSet.allOf(EnumSet.java:132)
	at com.intellij.ui.EnumComboBoxModel.createEnumSet(EnumComboBoxModel.java:24)
	at com.intellij.ui.EnumComboBoxModel.<init>(EnumComboBoxModel.java:19)
	at co.nums.intellij.aem.mH$a$4$5$1.l(Unknown Source)
	at co.nums.intellij.aem.mH$a$4$5$1.invoke(Unknown Source)
	at com.intellij.ui.dsl.builder.impl.PanelImpl.row(PanelImpl.kt:57)

[karollewandowski]

I'm sorry about the issue with missing bundle libraries. It is probably related to obfuscating an enum used in the configuration class. I've released a new version, which skips obfuscation for this enum, but I'm not sure whether it will fix it, or reconfiguration will be needed 😞 If it still doesn't work, I suggest reverting the plugin to the older version if it blocks your work.

Regarding the org.apache.felix.eventadmin:1.6.4 bundle, I noticed that one of the catch clauses in downloading method logged the exception on debug level, so maybe the error wasn't logged (I have changed it to warn in 2024.3.1-eap2; release in progress). If the issue is reproducible, you can add #co.nums.intellij.aem.BundleSources (with #) in Help | Diagnostic Tools | Debug Log Settings... and retry it, so it can be logged on debug level (there are also more information logged).

If there is still nothing in the logs, could you please attach here or send me via another channel this bundle file (it is open source and legal)? I will try to reproduce it on my side. My test AEM instance doesn't have the same bundle version.

[royteeuwen]

Here you go :)

2024-11-14 20:27:55,404 [ 308626]   FINE - #co.nums.intellij.aem.BundleSources - Sending request: GET https://repo1.maven.org/maven2/org/apache/felix/org.apache.felix.eventadmin/1.6.4/org.apache.felix.eventadmin-1.6.4-sources.jar HTTP/1.1
2024-11-14 20:27:55,563 [ 308785]   FINE - #co.nums.intellij.aem.BundleSources - Response: HttpResponseProxy{HTTP/1.1 200 OK [Connection: keep-alive, Content-Length: 57772, ETag: "34dc637f69513df109d2e386e8907e52", Content-Type: application/java-archive, Last-Modified: Tue, 26 Jul 2022 05:41:42 GMT, X-Checksum-MD5: 34dc637f69513df109d2e386e8907e52, X-Checksum-SHA1: 2f4c3b304dee200b6e0662c58b34e7c279ae59c7, Via: 1.1 varnish, 1.1 varnish, Accept-Ranges: bytes, Age: 92728, Date: Thu, 14 Nov 2024 19:27:55 GMT, X-Served-By: cache-iad-kiad7000145-IAD, cache-bru1480030-BRU, X-Cache: HIT, HIT, X-Cache-Hits: 13, 0, X-Timer: S1731612476.566429,VS0,VE1] ResponseEntityProxy{[Content-Type: application/java-archive,Content-Length: 57772,Chunked: false]}}
2024-11-14 20:27:55,565 [ 308787]   WARN - #co.nums.intellij.aem.BundleSources - Error occurred
java.io.FileNotFoundException: /Users/royteeuwen/.aem-ide/bundle-sources/org.apache.felix.eventadmin-1.6.4-sources.jar (No such file or directory)
	at java.base/java.io.FileOutputStream.open0(Native Method)
	at java.base/java.io.FileOutputStream.open(FileOutputStream.java:293)
	at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:235)
	at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:184)
	at co.nums.intellij.aem.mU.a(Unknown Source)
	at co.nums.intellij.aem.mU.b(Unknown Source)
	at co.nums.intellij.aem.mU.b(Unknown Source)
	at co.nums.intellij.aem.mK$c.run(Unknown Source)
	at com.intellij.openapi.progress.impl.CoreProgressManager.startTask(CoreProgressManager.java:477)
	at com.intellij.openapi.progress.impl.ProgressManagerImpl.startTask(ProgressManagerImpl.java:133)
	at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcessWithProgressAsynchronously$6(CoreProgressManager.java:528)
	at com.intellij.openapi.progress.impl.ProgressRunner.lambda$submit$4(ProgressRunner.java:250)
	at com.intellij.openapi.progress.ProgressManager.lambda$runProcess$0(ProgressManager.java:100)
	at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcess$1(CoreProgressManager.java:221)
	at com.intellij.platform.diagnostic.telemetry.helpers.TraceKt.use(trace.kt:46)
	at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcess$2(CoreProgressManager.java:220)
	at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$executeProcessUnderProgress$13(CoreProgressManager.java:660)
	at com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:735)
	at com.intellij.openapi.progress.impl.CoreProgressManager.computeUnderProgress(CoreProgressManager.java:691)
	at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:659)
	at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:79)
	at com.intellij.openapi.progress.impl.CoreProgressManager.runProcess(CoreProgressManager.java:202)
	at com.intellij.openapi.progress.ProgressManager.runProcess(ProgressManager.java:100)
	at com.intellij.openapi.progress.impl.ProgressRunner.lambda$submit$5(ProgressRunner.java:250)
	at com.intellij.openapi.progress.impl.ProgressRunner$ProgressRunnable.run(ProgressRunner.java:500)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:702)
	at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:699)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
	at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1.run(Executors.java:699)
	at java.base/java.lang.Thread.run(Thread.java:840)

[karollewandowski]

Thanks!

That was so lame bug. Fixed in 2024.3.1-eap3. Sorry for the inconvenience.

BTW, I assume that bundle libraries visibility got fixed after updating to 2024.3.1-eap2.

[royteeuwen]

I assume so as well :D I'm seeing the libraries again in any case ;)

[royteeuwen]

Would there be a way to move the fetching of the sources to a background process, so that I can continue working in the meantime?