Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sync on jcloudsLocation.vmInstanceIds #922

Merged
merged 1 commit into from
Dec 21, 2017
Merged

Sync on jcloudsLocation.vmInstanceIds #922

merged 1 commit into from
Dec 21, 2017

Conversation

aledsage
Copy link
Contributor

JcloudsMaxConcurrencyStubbedTest demonstrates the need for this synchronization: tearDown can fail because the obtain is called in a different thread from the releaseMachine. The id added to vmInstanceIds in obtain was not yet visible to the thread executing releaseMachine, so it failed.

To reproduce, try setting @Test(invocationCount=...) to something big. For me, it failed after 200ish runs.

2017-12-20 16:24:31,348 INFO  TESTNG INVOKING CONFIGURATION: "Surefire test" - @BeforeMethod org.apache.brooklyn.location.jclouds.JcloudsMaxConcurrencyStubbedTest.setUp()
2017-12-20 16:24:31,349 INFO  Added external config supplier named 'brooklyn-demo-sample': org.apache.brooklyn.core.config.external.InPlaceExternalConfigSupplier@67a84d80
2017-12-20 16:24:31,349 INFO  TESTNG PASSED CONFIGURATION: "Surefire test" - @BeforeMethod org.apache.brooklyn.location.jclouds.JcloudsMaxConcurrencyStubbedTest.setUp() finished in 1 ms
2017-12-20 16:24:31,349 INFO  TESTNG INVOKING: "Surefire test" - org.apache.brooklyn.location.jclouds.JcloudsMaxConcurrencyStubbedTest.testConcurrentCreateCalls()
2017-12-20 16:24:31,359 INFO  Creating VM aws-ec2:us-east-1 in JcloudsLocation[aws-ec2:us-east-1:stub-identity@r94r9yeb3n]
2017-12-20 16:24:31,360 WARN  Cannot check imageChooser status for aws-ec2:us-east-1 due to manually supplied black-box TemplateBuilder; it is recommended to use a PortableTemplateBuilder if you supply a TemplateBuilder
2017-12-20 16:24:31,362 INFO  Creating VM aws-ec2:us-east-1 in JcloudsLocation[aws-ec2:us-east-1:stub-identity@r94r9yeb3n]
2017-12-20 16:24:31,363 INFO  Creating VM aws-ec2:us-east-1 in JcloudsLocation[aws-ec2:us-east-1:stub-identity@r94r9yeb3n]
2017-12-20 16:24:31,363 WARN  Cannot check imageChooser status for aws-ec2:us-east-1 due to manually supplied black-box TemplateBuilder; it is recommended to use a PortableTemplateBuilder if you supply a TemplateBuilder
2017-12-20 16:24:31,363 INFO  Waiting in JcloudsLocation[aws-ec2:us-east-1:stub-identity@r94r9yeb3n] for machine-creation permit (0 other queuing requests already)
2017-12-20 16:24:31,366 INFO  Default SSH keys not found or not usable; will create new keys for each machine. Create ~/.ssh/id_rsa or set privateKeyFile / privateKeyPassphrase / password as appropriate for this location if you wish to be able to log in without Brooklyn.
2017-12-20 16:24:31,366 INFO  Default SSH keys not found or not usable; will create new keys for each machine. Create ~/.ssh/id_rsa or set privateKeyFile / privateKeyPassphrase / password as appropriate for this location if you wish to be able to log in without Brooklyn.
2017-12-20 16:24:31,523 INFO  Acquired in JcloudsLocation[aws-ec2:us-east-1:stub-identity@r94r9yeb3n] machine-creation permit, after waiting 160ms
2017-12-20 16:24:31,524 WARN  No context entity found in config or current task
2017-12-20 16:24:31,524 WARN  Cannot check imageChooser status for aws-ec2:us-east-1 due to manually supplied black-box TemplateBuilder; it is recommended to use a PortableTemplateBuilder if you supply a TemplateBuilder
2017-12-20 16:24:31,524 WARN  No context entity found in config or current task
2017-12-20 16:24:31,527 INFO  Default SSH keys not found or not usable; will create new keys for each machine. Create ~/.ssh/id_rsa or set privateKeyFile / privateKeyPassphrase / password as appropriate for this location if you wish to be able to log in without Brooklyn.
2017-12-20 16:24:31,545 INFO  Using host-and-port=173.194.32.123:22 and user=jenkins when connecting to {id=myid, status=RUNNING, loginPort=22, privateAddresses=[172.168.10.11], publicAddresses=[173.194.32.123], loginUser=myuser}
2017-12-20 16:24:31,545 INFO  Using host-and-port=173.194.32.123:22 and user=jenkins when connecting to {id=myid, status=RUNNING, loginPort=22, privateAddresses=[172.168.10.11], publicAddresses=[173.194.32.123], loginUser=myuser}
2017-12-20 16:24:31,552 INFO  Creating PortForwardManager(scope=global)
2017-12-20 16:24:31,555 WARN  Using DEPRECATED flag OPEN_IPTABLES (will not be supported in future versions) for SshMachineLocation[173.194.32.123:jenkins@173.194.32.123/173.194.32.123:22(id=dol3orvta0)] at JcloudsLocation[aws-ec2:us-east-1:stub-identity@r94r9yeb3n]
2017-12-20 16:24:31,556 WARN  Using DEPRECATED flag OPEN_IPTABLES (will not be supported in future versions) for SshMachineLocation[173.194.32.123:jenkins@173.194.32.123/173.194.32.123:22(id=fyilkxt1h4)] at JcloudsLocation[aws-ec2:us-east-1:stub-identity@r94r9yeb3n]
2017-12-20 16:24:31,558 INFO  Finished VM aws-ec2:us-east-1 creation: jenkins@173.194.32.123/173.194.32.123:22 ready after 196ms (semaphore obtained in 0ms;{image={id=us-east-1/bogus-image, providerId=ebs-image-provider, name=image, location={scope=REGION, id=us-east-1, description=us-east-1, parent=aws-ec2}, os={family=ubuntu, arch=paravirtual, version=1.0, description=ubuntu, is64Bit=true}, description=description, version=1.0, status=AVAILABLE, loginUser=root, userMetadata={rootDeviceType=ebs}}, hardware={id=supporting-bogus, providerId=t2.micro, processors=[{cores=1.0, speed=0.1}], ram=1024, supportsImage=Predicates.and(requiresRootDeviceType(ebs),requiresVirtualizationType(paravirtual),idIn([us-east-1/bogus-image]),Predicates.alwaysTrue())}, location={scope=REGION, id=us-east-1, description=us-east-1, parent=aws-ec2}, options={scriptPresent=true, userMetadata={Name=brooklyn-p19pkv-jenkins-ttta, brooklyn-user=jenkins}}} template built in 38ms; {id=myid, status=RUNNING, loginPort=22, privateAddresses=[172.168.10.11], publicAddresses=[173.194.32.123]} provisioned in 123ms; SshMachineLocation[173.194.32.123:jenkins@173.194.32.123/173.194.32.123:22(id=dol3orvta0)] connection usable in 25ms; and os customized in 10ms - point /dev/random to urandom, open iptables)
2017-12-20 16:24:31,561 INFO  Finished VM aws-ec2:us-east-1 creation: jenkins@173.194.32.123/173.194.32.123:22 ready after 201ms (semaphore obtained in 0ms;{image={id=us-east-1/bogus-image, providerId=ebs-image-provider, name=image, location={scope=REGION, id=us-east-1, description=us-east-1, parent=aws-ec2}, os={family=ubuntu, arch=paravirtual, version=1.0, description=ubuntu, is64Bit=true}, description=description, version=1.0, status=AVAILABLE, loginUser=root, userMetadata={rootDeviceType=ebs}}, hardware={id=supporting-bogus, providerId=t2.micro, processors=[{cores=1.0, speed=0.1}], ram=1024, supportsImage=Predicates.and(requiresRootDeviceType(ebs),requiresVirtualizationType(paravirtual),idIn([us-east-1/bogus-image]),Predicates.alwaysTrue())}, location={scope=REGION, id=us-east-1, description=us-east-1, parent=aws-ec2}, options={scriptPresent=true, userMetadata={Name=brooklyn-p19pkv-jenkins-ttta, brooklyn-user=jenkins}}} template built in 55ms; {id=myid, status=RUNNING, loginPort=22, privateAddresses=[172.168.10.11], publicAddresses=[173.194.32.123]} provisioned in 109ms; SshMachineLocation[173.194.32.123:jenkins@173.194.32.123/173.194.32.123:22(id=fyilkxt1h4)] connection usable in 25ms; and os customized in 12ms - point /dev/random to urandom, open iptables)
2017-12-20 16:24:31,564 WARN  No context entity found in config or current task
2017-12-20 16:24:31,580 INFO  Using host-and-port=173.194.32.123:22 and user=jenkins when connecting to {id=myid, status=RUNNING, loginPort=22, privateAddresses=[172.168.10.11], publicAddresses=[173.194.32.123], loginUser=myuser}
2017-12-20 16:24:31,588 WARN  Using DEPRECATED flag OPEN_IPTABLES (will not be supported in future versions) for SshMachineLocation[173.194.32.123:jenkins@173.194.32.123/173.194.32.123:22(id=mbzvjcfoql)] at JcloudsLocation[aws-ec2:us-east-1:stub-identity@r94r9yeb3n]
2017-12-20 16:24:31,590 INFO  Finished VM aws-ec2:us-east-1 creation: jenkins@173.194.32.123/173.194.32.123:22 ready after 228ms (semaphore obtained in 161ms;{image={id=us-east-1/bogus-image, providerId=ebs-image-provider, name=image, location={scope=REGION, id=us-east-1, description=us-east-1, parent=aws-ec2}, os={family=ubuntu, arch=paravirtual, version=1.0, description=ubuntu, is64Bit=true}, description=description, version=1.0, status=AVAILABLE, loginUser=root, userMetadata={rootDeviceType=ebs}}, hardware={id=supporting-bogus, providerId=t2.micro, processors=[{cores=1.0, speed=0.1}], ram=1024, supportsImage=Predicates.and(requiresRootDeviceType(ebs),requiresVirtualizationType(paravirtual),idIn([us-east-1/bogus-image]),Predicates.alwaysTrue())}, location={scope=REGION, id=us-east-1, description=us-east-1, parent=aws-ec2}, options={scriptPresent=true, userMetadata={Name=brooklyn-p19pkv-jenkins-sx1z, brooklyn-user=jenkins}}} template built in 39ms; {id=myid, status=RUNNING, loginPort=22, privateAddresses=[172.168.10.11], publicAddresses=[173.194.32.123]} provisioned in 1ms; SshMachineLocation[173.194.32.123:jenkins@173.194.32.123/173.194.32.123:22(id=mbzvjcfoql)] connection usable in 19ms; and os customized in 7ms - point /dev/random to urandom, open iptables)
2017-12-20 16:24:31,591 INFO  TESTNG PASSED: "Surefire test" - org.apache.brooklyn.location.jclouds.JcloudsMaxConcurrencyStubbedTest.testConcurrentCreateCalls() finished in 242 ms
2017-12-20 16:24:31,591 INFO  TESTNG INVOKING CONFIGURATION: "Surefire test" - @AfterMethod org.apache.brooklyn.location.jclouds.JcloudsMaxConcurrencyStubbedTest.tearDown()
2017-12-20 16:24:31,591 INFO  Releasing machine SshMachineLocation[173.194.32.123:jenkins@173.194.32.123/173.194.32.123:22(id=dol3orvta0)] in JcloudsLocation[aws-ec2:us-east-1:stub-identity@r94r9yeb3n], instance id myid
2017-12-20 16:24:31,594 INFO  Attempted release of unknown machine SshMachineLocation[173.194.32.123:jenkins@173.194.32.123/173.194.32.123:22(id=dol3orvta0)] in JcloudsLocation[aws-ec2:us-east-1:stub-identity@r94r9yeb3n]
2017-12-20 16:24:31,595 INFO  Released machine SshMachineLocation[173.194.32.123:jenkins@173.194.32.123/173.194.32.123:22(id=dol3orvta0)]: total time 3ms (semaphore obtained in 1ms; node destroyed in 0ms)
2017-12-20 16:24:31,595 INFO  Attempted release of unknown machine SshMachineLocation[173.194.32.123:jenkins@173.194.32.123/173.194.32.123:22(id=fyilkxt1h4)] in JcloudsLocation[aws-ec2:us-east-1:stub-identity@r94r9yeb3n]
2017-12-20 16:24:31,596 WARN  Error releasing machine SshMachineLocation[173.194.32.123:jenkins@173.194.32.123/173.194.32.123:22(id=fyilkxt1h4)]; continuing...
java.lang.IllegalArgumentException: Unknown machine SshMachineLocation[173.194.32.123:jenkins@173.194.32.123/173.194.32.123:22(id=fyilkxt1h4)]
	at org.apache.brooklyn.location.jclouds.JcloudsLocation.release(JcloudsLocation.java:2150) ~[classes/:na]
2017-12-20 16:24:31,597 INFO  Releasing machine SshMachineLocation[173.194.32.123:jenkins@173.194.32.123/173.194.32.123:22(id=mbzvjcfoql)] in JcloudsLocation[aws-ec2:us-east-1:stub-identity@r94r9yeb3n], instance id myid
2017-12-20 16:24:31,599 INFO  Attempted release of unknown machine SshMachineLocation[173.194.32.123:jenkins@173.194.32.123/173.194.32.123:22(id=mbzvjcfoql)] in JcloudsLocation[aws-ec2:us-east-1:stub-identity@r94r9yeb3n]
2017-12-20 16:24:31,600 INFO  Released machine SshMachineLocation[173.194.32.123:jenkins@173.194.32.123/173.194.32.123:22(id=mbzvjcfoql)]: total time 2ms (semaphore obtained in 0ms; node destroyed in 0ms)
2017-12-20 16:24:31,614 INFO  TESTNG FAILED CONFIGURATION: "Surefire test" - @AfterMethod org.apache.brooklyn.location.jclouds.JcloudsMaxConcurrencyStubbedTest.tearDown() finished in 0 ms
org.apache.brooklyn.util.exceptions.CompoundRuntimeException: Error in tearDown of class org.apache.brooklyn.location.jclouds.JcloudsMaxConcurrencyStubbedTest
	at org.apache.brooklyn.location.jclouds.AbstractJcloudsLiveTest.tearDown(AbstractJcloudsLiveTest.java:110)
	at org.apache.brooklyn.location.jclouds.AbstractJcloudsStubbedUnitTest.tearDown(AbstractJcloudsStubbedUnitTest.java:72)
	at org.apache.brooklyn.location.jclouds.JcloudsMaxConcurrencyStubbedTest.tearDown(JcloudsMaxConcurrencyStubbedTest.java:117)
Caused by: java.lang.IllegalArgumentException: Unknown machine SshMachineLocation[173.194.32.123:jenkins@173.194.32.123/173.194.32.123:22(id=fyilkxt1h4)]
	at org.apache.brooklyn.location.jclouds.JcloudsLocation.release(JcloudsLocation.java:2150)
	at org.apache.brooklyn.location.jclouds.AbstractJcloudsLiveTest.releaseMachine(AbstractJcloudsLiveTest.java:181)
	at org.apache.brooklyn.location.jclouds.AbstractJcloudsLiveTest.releaseMachineSafely(AbstractJcloudsLiveTest.java:189)
	at org.apache.brooklyn.location.jclouds.AbstractJcloudsLiveTest.tearDown(AbstractJcloudsLiveTest.java:91)
	... 33 more

@aledsage aledsage mentioned this pull request Dec 21, 2017
Merged
@tbouron
Copy link
Member

tbouron commented Dec 21, 2017

The build was aborted for some reason, but all tests passed:

[INFO] Installing org/apache/brooklyn/brooklyn-commands/1.0.0-SNAPSHOT/brooklyn-commands-1.0.0-SNAPSHOT.jar
[INFO] Writing OBR metadata
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
Build timed out (after 60 minutes). Marking the build as aborted.
Build was aborted
[JENKINS] Archiving disabled
[WARNING] Failed to notify spy hudson.maven.Maven3Builder$JenkinsEventSpy: java.lang.InterruptedException
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Brooklyn Server Root ............................... SUCCESS [  5.027 s]
[INFO] Brooklyn Parent Project ............................ SUCCESS [  4.852 s]
[INFO] Brooklyn Test Support Utilities .................... SUCCESS [  9.481 s]
[INFO] Brooklyn Logback Includable Configuration .......... SUCCESS [  3.821 s]
[INFO] Brooklyn Common Utilities .......................... SUCCESS [01:21 min]
[INFO] Brooklyn API ....................................... SUCCESS [  5.608 s]
[INFO] CAMP Server Parent Project ......................... SUCCESS [  2.409 s]
[INFO] CAMP Base .......................................... SUCCESS [  7.161 s]
[INFO] Brooklyn Test Support .............................. SUCCESS [  4.601 s]
[INFO] Brooklyn REST Swagger Apidoc Utilities ............. SUCCESS [  4.584 s]
[INFO] Brooklyn Logback Configuration ..................... SUCCESS [  3.591 s]
[INFO] CAMP Server ........................................ SUCCESS [  9.477 s]
[INFO] Brooklyn Felix Runtime ............................. SUCCESS [  7.076 s]
[INFO] Brooklyn Groovy Utilities .......................... SUCCESS [ 17.419 s]
[INFO] Brooklyn Core ...................................... SUCCESS [04:14 min]
[INFO] Brooklyn Policies .................................. SUCCESS [ 43.287 s]
[INFO] Brooklyn WinRM Software Entities ................... SUCCESS [ 17.993 s]
[INFO] Brooklyn Secure JMXMP Agent ........................ SUCCESS [ 14.597 s]
[INFO] Brooklyn JMX RMI Agent ............................. SUCCESS [  4.669 s]
[INFO] Brooklyn Jclouds Location Targets .................. SUCCESS [ 45.356 s]
[INFO] Brooklyn Software Base ............................. SUCCESS [01:46 min]
[INFO] Brooklyn CAMP ...................................... SUCCESS [04:02 min]
[INFO] Brooklyn Launcher Common ........................... SUCCESS [ 41.868 s]
[INFO] Brooklyn REST API .................................. SUCCESS [01:17 min]
[INFO] Brooklyn REST Resources ............................ SUCCESS [02:01 min]
[INFO] Brooklyn REST Server ............................... SUCCESS [01:56 min]
[INFO] Brooklyn Launcher .................................. SUCCESS [04:01 min]
[INFO] Brooklyn Container Location Targets ................ SUCCESS [01:58 min]
[INFO] Brooklyn Command Line Interface .................... SUCCESS [01:15 min]
[INFO] Brooklyn Test Framework ............................ SUCCESS [02:11 min]
[INFO] Brooklyn OSGi init ................................. SUCCESS [01:34 min]
[INFO] Brooklyn OSGi start ................................ SUCCESS [ 59.515 s]
[INFO] Brooklyn Karaf ..................................... SUCCESS [ 25.970 s]
[INFO] Jetty config fragment .............................. SUCCESS [03:20 min]
[INFO] Apache Http Component extension .................... SUCCESS [02:21 min]
[INFO] Brooklyn Karaf Features ............................ SUCCESS [ 52.768 s]
[INFO] Brooklyn Karaf Shell Commands ...................... SUCCESS [02:49 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 59:14 min
[INFO] Finished at: 2017-12-21T12:22:15+00:00
[INFO] Final Memory: 99M/801M
[INFO] ------------------------------------------------------------------------
channel stopped
Adding one-line test results to commit status...
Setting status of 8670b23b075f944b27d91e0075bd0f4af1ba9175 to FAILURE with url https://builds.apache.org/job/brooklyn-server-pull-requests/2818/ and message: 'FAILURE
 4961 tests run, 0 skipped, 0 failed.'
Using context: Jenkins: brooklyn-server-pull-request
Finished: ABORTED

@tbouron
Copy link
Member

tbouron commented Dec 21, 2017

retest this please

@tbouron
Copy link
Member

tbouron commented Dec 21, 2017

LGTM, will merge once Jenkins is happy

@tbouron
Copy link
Member

tbouron commented Dec 21, 2017

retest this please

@aledsage
Copy link
Contributor Author

Thanks @tbouron - jenkins is happy so merging.

@asfgit asfgit merged commit 8670b23 into apache:master Dec 21, 2017
asfgit pushed a commit that referenced this pull request Dec 21, 2017
@aledsage
This closes #922
274a930
@aledsage aledsage deleted the fix-JcloudsMaxConcurrencyStubbedTest branch February 5, 2018 09:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
3 participants
@aledsage @tbouron @asfgit