[OODT-978] Fix for Crawler Launcher hanging even after ingesting all the files (when avro fm client is used) #62

IMS94 · 2018-05-16T10:32:34Z

No description provided.

…implementations.

…version by adding a close() method to the FileManagerClient

Replaced with RpcCommunicationsFactory.createClient()

chrismattmann · 2018-05-17T17:26:33Z

LGTM!

chrismattmann · 2018-05-17T17:27:29Z

testing this now locally

chrismattmann · 2018-05-17T18:05:40Z

@IMS94 I get an error building this:

INFO: Loading workflow context metadata...
Tests run: 15, Failures: 0, Errors: 1, Skipped: 1, Time elapsed: 1.242 sec <<< FAILURE! - in org.apache.oodt.cas.pge.TestPGETaskInstance
testCreatePgeConfig(org.apache.oodt.cas.pge.TestPGETaskInstance)  Time elapsed: 0.01 sec  <<< ERROR!
java.io.IOException: Failed to build PgeConfig : Failed to parse value: /var/folders/n5/1d_k3z4s2293q8ntx_n8sw54mm5n_8/T/3ea2e425-c368-4fc3-b73b-fe45724a17b5/pgeConfig.xml
	at java.net.URL.<init>(URL.java:532)
	at java.net.URL.<init>(URL.java:490)
	at java.net.URL.<init>(URL.java:439)
	at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:444)
	at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:438)
	at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeConfigBuilder.java:58)
	at org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInstance.java:218)
	at org.apache.oodt.cas.pge.TestPGETaskInstance.testCreatePgeConfig(TestPGETaskInstance.java:260)

Running org.apache.oodt.cas.pge.metadata.TestPgeMetadata
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in org.apache.oodt.cas.pge.metadata.TestPgeMetadata
Running org.apache.oodt.cas.pge.metadata.TestPgeTaskMetKeys
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in org.apache.oodt.cas.pge.metadata.TestPgeTaskMetKeys

Results :

Tests in error: 
  TestXmlHelper.testGetImports:52 » PGE Failed to parse value: blank
  TestXmlHelper.testGetNamespace:65 » PGE Failed to parse value: blank
  TestXmlFilePgeConfigBuilder.testBuild:54 » IO Failed to build PgeConfig : Fail...
  TestPGETaskInstance.testCreatePgeConfig:260 » IO Failed to build PgeConfig : F...

Tests run: 31, Failures: 0, Errors: 4, Skipped: 1

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] OODT Core .......................................... SUCCESS [  3.146 s]
[INFO] Common Utilities ................................... SUCCESS [ 27.736 s]
[INFO] CAS Command Line Interface ......................... SUCCESS [  6.426 s]
[INFO] OODT - Configuration Management .................... SUCCESS [ 23.711 s]
[INFO] Process Control System Input Data Package .......... SUCCESS [  2.284 s]
[INFO] Catalog and Archive Service Generic Multi-valued Metadata Container SUCCESS [  3.407 s]
[INFO] Catalog and Archive File Management Component ...... SUCCESS [03:50 min]
[INFO] Catalog and Archive Resource Management Component .. SUCCESS [ 33.547 s]
[INFO] Catalog and Archive Workflow Management Component .. SUCCESS [ 42.013 s]
[INFO] Catalog and Archive Crawling Framework ............. SUCCESS [ 26.329 s]
[INFO] OODT CAS Curator Single Sign On Security Package ... SUCCESS [  2.354 s]
[INFO] CAS Curation Web Services .......................... SUCCESS [ 11.634 s]
[INFO] Process Control System Core Package ................ SUCCESS [ 20.501 s]
[INFO] OODT Wicket Web Components ......................... SUCCESS [  6.201 s]
[INFO] CAS Curation Interface ............................. SUCCESS [ 18.657 s]
[INFO] CAS PGE Adaptor Framework .......................... FAILURE [  7.536 s]
[INFO] CAS Installer Maven Mojo ........................... SKIPPED
[INFO] OODT :: Archetypes :: OpsUI ........................ SKIPPED
[INFO] OODT :: Archetypes :: RADiX ........................ SKIPPED
[INFO] OODT :: Archetypes ................................. SKIPPED
[INFO] CAS File Manager Browser Web App ................... SKIPPED
[INFO] CAS Workflow Manager Monitor Web App ............... SKIPPED
[INFO] CAS Product Server Web Application ................. SKIPPED
[INFO] CAS Workflow REST Services ......................... SKIPPED
[INFO] Process Control System Operator Interface Webapp ... SKIPPED
[INFO] OODT Process Control System JAX-RS service layer ... SKIPPED
[INFO] Apache OODT ........................................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 07:46 min
[INFO] Finished at: 2018-05-17T10:35:12-07:00
[INFO] Final Memory: 116M/1554M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on project cas-pge: There are test failures.
[ERROR] 
[ERROR] Please refer to /Users/mattmann/git/oodt/pge/target/surefire-reports for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :cas-pge
nonas:oodt mattmann$

Any ideas?

IMS94 · 2018-05-18T03:05:01Z

@chrismattmann I will look at that. Seems like this is not related to my changes. Let's see.

chrismattmann · 2018-05-18T04:57:38Z

@IMS94 I think you are right. In fact I think it's related to @lewismc and the update for OODT-976 in #60

chrismattmann · 2018-05-18T05:24:41Z

@IMS94 OK here's what I did. This is definitely something related to your PR/patch:

  517  git reset --hard 3cac96e4d51c19e0fa2467be3cb66350bc2dfe01
  518  mvn clean install
  519  git apply fed87b2bccc9890b0f5e45883936cc31fcd12f8f
  520  git cherry-pick fed87b2bccc9890b0f5e45883936cc31fcd12f8f
  521  mvn clean install
  522  git cherry-pick 40ed14e48510b9e6993caaa97c7d69bad9ff99a7
  523  mvn clean install
  524  git cherry-pick 1c49a9bbbc8e7c95bd23106c3dc58560c5af7488
  525  git cherry-pick 2d19275c1f142d4d42f9765aad8904ebbe7e5a12
  526  mvn clean install

Output from my history. That resets to the DOAP file commit pre commits from @lewismc and you. Then cherry picks and applies commits without merge commits. You can see that it breaks after I apply !525:

INFO: Loading workflow context metadata...
Tests run: 15, Failures: 0, Errors: 1, Skipped: 1, Time elapsed: 1.24 sec <<< FAILURE! - in org.apache.oodt.cas.pge.TestPGETaskInstance
testCreatePgeConfig(org.apache.oodt.cas.pge.TestPGETaskInstance)  Time elapsed: 0.007 sec  <<< ERROR!
java.io.IOException: Failed to build PgeConfig : Failed to parse value: /var/folders/n5/1d_k3z4s2293q8ntx_n8sw54mm5n_8/T/904bacbf-0121-47a9-981a-a0a5db5e0e85/pgeConfig.xml
	at java.net.URL.<init>(URL.java:532)
	at java.net.URL.<init>(URL.java:490)
	at java.net.URL.<init>(URL.java:439)
	at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:444)
	at org.apache.oodt.cas.pge.util.XmlHelper.fillIn(XmlHelper.java:438)
	at org.apache.oodt.cas.pge.config.XmlFilePgeConfigBuilder.build(XmlFilePgeConfigBuilder.java:58)
	at org.apache.oodt.cas.pge.PGETaskInstance.createPgeConfig(PGETaskInstance.java:218)
	at org.apache.oodt.cas.pge.TestPGETaskInstance.testCreatePgeConfig(TestPGETaskInstance.java:260)

Running org.apache.oodt.cas.pge.metadata.TestPgeMetadata
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in org.apache.oodt.cas.pge.metadata.TestPgeMetadata
Running org.apache.oodt.cas.pge.metadata.TestPgeTaskMetKeys
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in org.apache.oodt.cas.pge.metadata.TestPgeTaskMetKeys

Results :

Tests in error: 
  TestXmlHelper.testGetImports:52 » PGE Failed to parse value: blank
  TestXmlHelper.testGetNamespace:65 » PGE Failed to parse value: blank
  TestXmlFilePgeConfigBuilder.testBuild:54 » IO Failed to build PgeConfig : Fail...
  TestPGETaskInstance.testCreatePgeConfig:260 » IO Failed to build PgeConfig : F...

Tests run: 31, Failures: 0, Errors: 4, Skipped: 1

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] OODT Core .......................................... SUCCESS [  2.580 s]
[INFO] Common Utilities ................................... SUCCESS [ 25.469 s]
[INFO] CAS Command Line Interface ......................... SUCCESS [ 10.728 s]
[INFO] OODT - Configuration Management .................... SUCCESS [ 28.755 s]
[INFO] Process Control System Input Data Package .......... SUCCESS [  4.786 s]
[INFO] Catalog and Archive Service Generic Multi-valued Metadata Container SUCCESS [  4.022 s]
[INFO] Catalog and Archive File Management Component ...... SUCCESS [04:25 min]
[INFO] Catalog and Archive Resource Management Component .. SUCCESS [ 37.578 s]
[INFO] Catalog and Archive Workflow Management Component .. SUCCESS [ 40.678 s]
[INFO] Catalog and Archive Crawling Framework ............. SUCCESS [ 28.660 s]
[INFO] OODT CAS Curator Single Sign On Security Package ... SUCCESS [  1.679 s]
[INFO] CAS Curation Web Services .......................... SUCCESS [ 11.258 s]
[INFO] Process Control System Core Package ................ SUCCESS [ 21.118 s]
[INFO] OODT Wicket Web Components ......................... SUCCESS [  5.078 s]
[INFO] CAS Curation Interface ............................. SUCCESS [ 15.037 s]
[INFO] CAS PGE Adaptor Framework .......................... FAILURE [  6.886 s]
[INFO] CAS Installer Maven Mojo ........................... SKIPPED
[INFO] OODT :: Archetypes :: OpsUI ........................ SKIPPED
[INFO] OODT :: Archetypes :: RADiX ........................ SKIPPED
[INFO] OODT :: Archetypes ................................. SKIPPED
[INFO] CAS File Manager Browser Web App ................... SKIPPED
[INFO] CAS Workflow Manager Monitor Web App ............... SKIPPED
[INFO] CAS Product Server Web Application ................. SKIPPED
[INFO] CAS Workflow REST Services ......................... SKIPPED
[INFO] Process Control System Operator Interface Webapp ... SKIPPED
[INFO] OODT Process Control System JAX-RS service layer ... SKIPPED
[INFO] Apache OODT ........................................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 08:30 min
[INFO] Finished at: 2018-05-17T22:20:02-07:00
[INFO] Final Memory: 126M/1517M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on project cas-pge: There are test failures.
[ERROR] 
[ERROR] Please refer to /Users/mattmann/git/oodt/pge/target/surefire-reports for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :cas-pge
nonas:oodt mattmann$

…r.fillIn()

IMS94 · 2018-05-18T08:02:46Z

@chrismattmann Can you check now?

chrismattmann · 2018-05-18T15:46:49Z

nice one @IMS94 I will pull and test now.

chrismattmann · 2018-05-18T16:02:31Z

Works great now, I'm going to merge it!

[INFO] --- maven-javadoc-plugin:2.10.3:javadoc (attach-javadocs) @ oodt ---
[INFO] 
[INFO] --- maven-install-plugin:2.5.2:install (default-install) @ oodt ---
[INFO] Installing /Users/mattmann/git/oodt/pom.xml to /Users/mattmann/.m2/repository/org/apache/oodt/oodt/1.9-SNAPSHOT/oodt-1.9-SNAPSHOT.pom
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] OODT Core .......................................... SUCCESS [  4.124 s]
[INFO] Common Utilities ................................... SUCCESS [ 30.957 s]
[INFO] CAS Command Line Interface ......................... SUCCESS [ 12.048 s]
[INFO] OODT - Configuration Management .................... SUCCESS [ 34.007 s]
[INFO] Process Control System Input Data Package .......... SUCCESS [  3.404 s]
[INFO] Catalog and Archive Service Generic Multi-valued Metadata Container SUCCESS [  4.103 s]
[INFO] Catalog and Archive File Management Component ...... SUCCESS [03:58 min]
[INFO] Catalog and Archive Resource Management Component .. SUCCESS [ 31.632 s]
[INFO] Catalog and Archive Workflow Management Component .. SUCCESS [ 41.087 s]
[INFO] Catalog and Archive Crawling Framework ............. SUCCESS [ 28.410 s]
[INFO] OODT CAS Curator Single Sign On Security Package ... SUCCESS [  1.701 s]
[INFO] CAS Curation Web Services .......................... SUCCESS [ 11.687 s]
[INFO] Process Control System Core Package ................ SUCCESS [ 19.636 s]
[INFO] OODT Wicket Web Components ......................... SUCCESS [  6.191 s]
[INFO] CAS Curation Interface ............................. SUCCESS [ 16.762 s]
[INFO] CAS PGE Adaptor Framework .......................... SUCCESS [ 27.703 s]
[INFO] CAS Installer Maven Mojo ........................... SUCCESS [  6.616 s]
[INFO] OODT :: Archetypes :: OpsUI ........................ SUCCESS [  2.411 s]
[INFO] OODT :: Archetypes :: RADiX ........................ SUCCESS [  1.142 s]
[INFO] OODT :: Archetypes ................................. SUCCESS [  0.025 s]
[INFO] CAS File Manager Browser Web App ................... SUCCESS [ 13.762 s]
[INFO] CAS Workflow Manager Monitor Web App ............... SUCCESS [  6.312 s]
[INFO] CAS Product Server Web Application ................. SUCCESS [ 13.493 s]
[INFO] CAS Workflow REST Services ......................... SUCCESS [  5.431 s]
[INFO] Process Control System Operator Interface Webapp ... SUCCESS [ 31.500 s]
[INFO] OODT Process Control System JAX-RS service layer ... SUCCESS [  8.884 s]
[INFO] Apache OODT ........................................ SUCCESS [  0.021 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10:02 min
[INFO] Finished at: 2018-05-18T08:58:23-07:00
[INFO] Final Memory: 136M/1566M
[INFO] ------------------------------------------------------------------------
nonas:oodt mattmann$

IMS94 added 6 commits April 18, 2018 18:45

Updated startup scripts of resmgr and workflowmanager to use AvroRpc …

ecfe0a7

…implementations.

Merge remote-tracking branch 'origin/master'

3fcd424

Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/oodt

19e5162

[OODT-978] Fixed crawler launcher hanging issue in file manager avro …

1c49a9b

…version by adding a close() method to the FileManagerClient

Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/oodt

dc79494

Completely removed XmlRpcFileManager from all the unnecessary locations.

2d19275

Replaced with RpcCommunicationsFactory.createClient()

Fixed TestPGETaskInstance test failure due to null FM Url at XMLHelpe…

ec25440

…r.fillIn()

asfgit merged commit ec25440 into apache:master May 18, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[OODT-978] Fix for Crawler Launcher hanging even after ingesting all the files (when avro fm client is used) #62

[OODT-978] Fix for Crawler Launcher hanging even after ingesting all the files (when avro fm client is used) #62

Uh oh!

IMS94 commented May 16, 2018

Uh oh!

chrismattmann commented May 17, 2018

Uh oh!

chrismattmann commented May 17, 2018

Uh oh!

chrismattmann commented May 17, 2018

Uh oh!

IMS94 commented May 18, 2018

Uh oh!

chrismattmann commented May 18, 2018

Uh oh!

chrismattmann commented May 18, 2018

Uh oh!

IMS94 commented May 18, 2018

Uh oh!

chrismattmann commented May 18, 2018

Uh oh!

chrismattmann commented May 18, 2018

Uh oh!

Uh oh!

[OODT-978] Fix for Crawler Launcher hanging even after ingesting all the files (when avro fm client is used) #62

[OODT-978] Fix for Crawler Launcher hanging even after ingesting all the files (when avro fm client is used) #62

Uh oh!

Conversation

IMS94 commented May 16, 2018

Uh oh!

chrismattmann commented May 17, 2018

Uh oh!

chrismattmann commented May 17, 2018

Uh oh!

chrismattmann commented May 17, 2018

Uh oh!

IMS94 commented May 18, 2018

Uh oh!

chrismattmann commented May 18, 2018

Uh oh!

chrismattmann commented May 18, 2018

Uh oh!

IMS94 commented May 18, 2018

Uh oh!

chrismattmann commented May 18, 2018

Uh oh!

chrismattmann commented May 18, 2018

Uh oh!

Uh oh!