Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Require httpclientjava instead of httpclient5 #206

Merged
merged 1 commit into from
Sep 19, 2023

Conversation

laeubi
Copy link
Member

@laeubi laeubi commented Dec 19, 2022

Fix #205

Copy link
Member

@akurtakov akurtakov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change looks good but I haven't tested it at all. I am pretty sure that the only way for this to be thoroughly tested is via I-build though.

@laeubi
Copy link
Member Author

laeubi commented Dec 19, 2022

The change looks good but I haven't tested it at all. I am pretty sure that the only way for this to be thoroughly tested is via I-build though.

Yes I'm also not sure how P2 will behave here... as it is only a prerequisite and not a (strict) requirement it might just install the new connector additionally?

I think if the test suceed, we can merge and must see what happens...

@vogella
Copy link
Contributor

vogella commented Dec 19, 2022

AFAIK testing p2 from a runtime Eclipse is not really possible so +1 for merging to see it used in an I-build (if the validation build is green).

@laeubi
Copy link
Member Author

laeubi commented Dec 19, 2022

Seems it is not that easy... have to investigate further, but if someone else likes to take a look help would be welcome.

Copy link
Contributor

@merks merks left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was able to install a feature (EMFs SDK feature) in a self-hosted runtime where software installation is enabled. So it appears to work well for me.

@merks
Copy link
Contributor

merks commented Dec 20, 2022

Note that in my self-host launch running the org.eclipse.sdk.ide product, the following ECF bundles are installed:

image

The expected new provider is installed...

@vogella
Copy link
Contributor

vogella commented Mar 30, 2023

Time to merge this one?

Copy link
Contributor

@vogella vogella left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 for merging to see this in a running IDE. If we face issues we still can revert

@HannesWell
Copy link
Member

+1 for merging to see this in a running IDE. If we face issues we still can revert

I cannot assess the change, but if it is risky then we should it better sooner than later.

@akurtakov
Copy link
Member

It needs someone to sign up to keep an eye on it and it has to go for M1 if at all. I can land a hand but can't be the driving force here. So if someone feels like it today is the day or it would miss yet another release.

@laeubi
Copy link
Member Author

laeubi commented Mar 31, 2023

I have rebased the change now, but I won't have time too look into the testfailures or give dedicated support for it.

@iloveeclipse
Copy link
Member

It needs someone to sign up to keep an eye on it and it has to go for M1 if at all. I can land a hand but can't be the driving force here. So if someone feels like it today is the day or it would miss yet another release.

I can watch for issues, and offer to revert it ASAP if needed :-)

@merks
Copy link
Contributor

merks commented Mar 31, 2023

It looks like all the builds so far are aborting:

https://ci.eclipse.org/equinox/job/p2/job/PR-206/

[ERROR] Failed to execute goal org.eclipse.tycho:tycho-surefire-plugin:4.0.0-SNAPSHOT:test (default-test) on project org.eclipse.equinox.p2.tests: An unexpected error occurred while launching the test runtime (process returned error code 143(SIGTERM received?)).

Without timestamps in the log, one can't really tell which parts are taking so long, but each seems to abort during the p2 tests...

Normal builds appear to take about 20 minutes:

https://ci.eclipse.org/equinox/job/p2/job/master

Similarly the p2 action build is taking very long:

image

While a normal such build should take about 15 minutes:

image

Something is either very slow or non-terminating...

@laeubi
Copy link
Member Author

laeubi commented Mar 31, 2023

Yes I assume some test is waiting for "something" to happen and then blocks forever, but I haven't had time yet to investigate this.

@akurtakov
Copy link
Member

I would be very happey if this one lands in now. @laeubi do you plan to do so?

@laeubi
Copy link
Member Author

laeubi commented Jun 8, 2023

I have rebased the change lets see what happens but last time the tests started to hang infinite...

@laeubi
Copy link
Member Author

laeubi commented Jun 8, 2023

One of the test fail here:

org.eclipse.equinox.p2.core.ProvisionException: Unable to read repository at https://download.eclipse.org/eclipse/updates/4.21/R-4.21-202109060500/content.xml.xz.
	at org.eclipse.equinox.internal.p2.repository.CacheManager.createCacheFromFile(CacheManager.java:132)
	at org.eclipse.equinox.internal.p2.metadata.repository.XZedSimpleMetadataRepositoryFactory.getLocalFile(XZedSimpleMetadataRepositoryFactory.java:60)
	at org.eclipse.equinox.internal.p2.metadata.repository.XZedSimpleMetadataRepositoryFactory.load(XZedSimpleMetadataRepositoryFactory.java:80)
	at org.eclipse.equinox.internal.p2.metadata.repository.MetadataRepositoryManager.factoryLoad(MetadataRepositoryManager.java:63)
	at org.eclipse.equinox.internal.p2.repository.helpers.AbstractRepositoryManager.loadRepository(AbstractRepositoryManager.java:787)
	at org.eclipse.equinox.internal.p2.repository.helpers.AbstractRepositoryManager.loadRepository(AbstractRepositoryManager.java:685)
	at org.eclipse.equinox.internal.p2.metadata.repository.MetadataRepositoryManager.loadRepository(MetadataRepositoryManager.java:110)
	at org.eclipse.equinox.internal.p2.metadata.repository.MetadataRepositoryManager.loadRepository(MetadataRepositoryManager.java:105)
	at org.eclipse.equinox.p2.tests.metadata.repository.MetadataRepositoryManagerTest.testMetadataCachingRemoteRepo(MetadataRepositoryManagerTest.java:396)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at junit.framework.TestCase.runTest(TestCase.java:177)
	at org.eclipse.equinox.p2.tests.AbstractProvisioningTest.runTest(AbstractProvisioningTest.java:968)
	at junit.framework.TestCase.runBare(TestCase.java:142)
	at junit.framework.TestResult$1.protect(TestResult.java:122)
	at junit.framework.TestResult.runProtected(TestResult.java:142)
	at junit.framework.TestResult.run(TestResult.java:125)
	at junit.framework.TestCase.run(TestCase.java:130)
	at junit.framework.TestSuite.runTest(TestSuite.java:241)
	at junit.framework.TestSuite.run(TestSuite.java:236)
	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:90)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:316)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:240)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:214)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:155)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.apache.maven.surefire.api.util.ReflectionUtils.invokeMethodWithArray2(ReflectionUtils.java:137)
	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:148)
	at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:88)
	at org.eclipse.tycho.surefire.osgibooter.OsgiSurefireBooter.run(OsgiSurefireBooter.java:140)
	at org.eclipse.tycho.surefire.osgibooter.HeadlessTestApplication.start(HeadlessTestApplication.java:29)
	at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:203)
	at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:136)
	at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:402)
	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:255)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:659)
	at org.eclipse.equinox.launcher.Main.basicRun(Main.java:596)
	at org.eclipse.equinox.launcher.Main.run(Main.java:1467)
	at org.eclipse.equinox.launcher.Main.main(Main.java:1440)
Caused by: java.io.IOException: Invalid last modified value
	at org.eclipse.ecf.provider.filetransfer.httpclientjava.HttpClientFileSystemBrowser.getLastModifiedTimeFromHeader(HttpClientFileSystemBrowser.java:335)
	at org.eclipse.ecf.provider.filetransfer.httpclientjava.HttpClientFileSystemBrowser.runRequest(HttpClientFileSystemBrowser.java:277)
	at org.eclipse.ecf.provider.filetransfer.browse.AbstractFileSystemBrowser$DirectoryJob.run(AbstractFileSystemBrowser.java:71)
	at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)

The URl in question returns Last-Modified: Sat, 16 Mar 2019 16:14:09 GMT but the string looks fine for me...

@akurtakov
Copy link
Member

Sounds like something to be improved in ecf httpclient .

@laeubi
Copy link
Member Author

laeubi commented Jun 8, 2023

I have created

so it should work with an english locale, when execute the build I get some more errors that seem to be related to locales (but not to the transport) that is a bit unfortunate as it seems that

  1. tests assume a certain locale to succeed
  2. some test fail with a different locale so we can't really test with different ones...

@laeubi
Copy link
Member Author

laeubi commented Jun 8, 2023

I fear we need a new ECF release first before we can proceed here but, It would be good if someone else could also execute the build locally (a simple mvn clean verify should suffice) and check if they see any other issues yet.

@laeubi
Copy link
Member Author

laeubi commented Sep 18, 2023

@scottslewis
Copy link

Alright we have no longer a hang of the test but one test-failure that looks a bit strange:

https://ci.eclipse.org/equinox/job/p2/job/PR-206/9/testReport/junit/org.eclipse.equinox.p2.tests.artifact.repository/TransferTest/testGZFileAreNotUnzipped/ https://github.com/eclipse-equinox/p2/pull/206/checks?check_run_id=16891673579

This does not appear to me to have anything to do with ECF or the javahttpclient provider.

java.lang.NullPointerException: Cannot invoke "org.osgi.framework.Bundle.getSymbolicName()" because "bundle" is null

It seems strange to me that Bundle.getSymbolicName() would ever fail in the Eclipse testing environment but perhaps it's possible under some conditions?

@laeubi
Copy link
Member Author

laeubi commented Sep 19, 2023

It seems strange to me that Bundle.getSymbolicName() would ever fail in the Eclipse testing environment but perhaps it's possible under some conditions?

the bundle itself is null so maybe something has changed here where a specific bundle name is assumed, I'll take a look... I just wanted to point out that now tests are all running (even though not passing) that before where blocked forever so that is now fixed 👍

@laeubi
Copy link
Member Author

laeubi commented Sep 19, 2023

The problem is this line:

https://github.com/eclipse-equinox/p2/blob/60a4a7ee8d5f15d1c1a7e45406efe12edbe82f63/bundles/org.eclipse.equinox.p2.tests/src/org/eclipse/equinox/p2/tests/artifact/repository/TransferTest.java#L62C1-L62C7

It seems only a debugline so I'll adjust it, beside that the test seem to reference packed artifacts so probably should be removed at all @akurtakov ?

@akurtakov
Copy link
Member

The test referes to pack.gz file but this seems to be because this was the first gziped file found convenient. It could actually be any gzipped file.

@laeubi
Copy link
Member Author

laeubi commented Sep 19, 2023

@vogella @merks @akurtakov the build now succeeds, I'll leave it up to you to merge this if desired.

@akurtakov
Copy link
Member

In the current shape there are high chances to fail the build as I'm not sure that all the products will find content from the extra p2 repo added here.
@scottslewis Would you please release ecf so we can update to it in Platform's target platform?
After new ecf version is in target platform this one would have to be amended and the merge can happen.

@mickaelistria
Copy link
Contributor

In the meantime, couldn't we update the .target to use the snapshots of ECF and then merge this patch without the extra repo and keep a ticket open to ensure we update to ECF release ASAP ?

@akurtakov
Copy link
Member

If the snapshot is here to stay till a release - yes.

@mickaelistria
Copy link
Contributor

Looks like the https://download.eclipse.org/rt/ecf/snapshot/site.p2/ contains snapshots which are more than 1 year old, so let's just use it in .target until there is a release ( @scottslewis the sooner the better ;) ).
I'm taking care of the change in .target.

@mickaelistria
Copy link
Contributor

See eclipse-platform/eclipse.platform.releng.aggregator#1372 about updating the .target.

@akurtakov
Copy link
Member

@mickaelistria thanks!

mickaelistria added a commit to mickaelistria/eclipse.platform.releng.aggregator that referenced this pull request Sep 19, 2023
So we can start consuming the new httpclientjava and getting rid of
httpclient5.

The snapshot repo must be replaced by release one as soon as ECF
provides a release, before Eclipse 4.30.M3.

See eclipse-equinox/p2#206
mickaelistria added a commit to eclipse-platform/eclipse.platform.releng.aggregator that referenced this pull request Sep 19, 2023
So we can start consuming the new httpclientjava and getting rid of
httpclient5.

The snapshot repo must be replaced by release one as soon as ECF
provides a release, before Eclipse 4.30.M3.

See eclipse-equinox/p2#206
@mickaelistria
Copy link
Contributor

The ECF snapshot was added to the .target. Can you please amend the commit to remove the repository entry?

@laeubi
Copy link
Member Author

laeubi commented Sep 19, 2023

Done.

@mickaelistria
Copy link
Contributor

Build fails with

Missing requirement: org.eclipse.equinox.p2.core.feature.feature.group 1.7.0.qualifier requires 'org.eclipse.equinox.p2.iu; org.eclipse.ecf.filetransfer.httpclientjava.feature.feature.group [1.0.0,2.0.0)' but it could not be found

In the .target with eclipse-platform/eclipse.platform.releng.aggregator#1372 , I've used <unit id="org.eclipse.ecf.filetransfer.httpclientjava.feature.feature.group" version="2.0.0.v20230715-1752"/>. Can you please change the version of the versionrange of the requirement accordingly?

@mickaelistria mickaelistria merged commit 3867228 into eclipse-equinox:master Sep 19, 2023
8 checks passed
@mickaelistria
Copy link
Contributor

Merged. Thanks a lot!
For the moment, I've left httpclient5 part of the .target; I hope we will be able to remove it after next I-Build, or are there any other known client left in Platform?

@iloveeclipse
Copy link
Member

Would be nice to trigger an I Build now to verify if anything is broken.

@mickaelistria
Copy link
Contributor

OK, I'm triggering I-Build.

@merks
Copy link
Contributor

merks commented Sep 19, 2023

Merged. Thanks a lot! For the moment, I've left httpclient5 part of the .target; I hope we will be able to remove it after next I-Build, or are there any other known client left in Platform?

@mickaelistria if you like, I'll add this cleanup to my TODO list. I've already done some investigation previously for the impact on Oomph which is also directly using ECF. Here too the switch is mostly painless...

@mickaelistria
Copy link
Contributor

OK, I happily leave it the continuation of the cleanup to you @merks !

@scottslewis
Copy link

Looks like the https://download.eclipse.org/rt/ecf/snapshot/site.p2/ contains snapshots which are more than 1 year old, so let's just use it in .target until there is a release ( @scottslewis the sooner the better ;) ). I'm taking care of the change in .target.

I would prefer to wait one week for a full release of ECF, as a full release requires some extra releng that I don't have time for this week. Also, IMO it would be nice to have one platform integration release for a 'smoke test' of javahttpclient provider (and perhaps more difficult removing the httpclient5 provider and all org.apache.* deps) before I do a release as I don't see doing weekly releases of ECF as feasible for me right now.

It seems that maybe you've already used the latest at https://download.eclipse.org/rt/ecf/snapshot/site.p2/, which is:

https://download.eclipse.org/rt/ecf/snapshot/site.p2/3.14.40.v20230905-1933/

in eclipse-platform/eclipse.platform.releng.aggregator#1372

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Switch to Java11 HttpClient
8 participants