Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix regression test failures #5

Open
10 of 15 tasks
liblit opened this issue Dec 29, 2017 · 0 comments
Open
10 of 15 tasks

Fix regression test failures #5

liblit opened this issue Dec 29, 2017 · 0 comments
Labels
bug help wanted
Milestone
Gradle and Buildship

Comments

@liblit
Copy link
Owner

liblit commented Dec 29, 2017 •
edited

Use ./gradelw test to run WALA’s automated regression tests. Several test suite subprojects pass completely. However, at least the following fail:

  • com.ibm.wala.cast.js.rhino.test
  • com.ibm.wala.cast.js.test
  • com.ibm.wala.core.tests
    • DynamicCallGraphTest
    • GetTargetsTest
    • PruneArrayOutOfBoundExceptionEdge
  • com.ibm.wala.dalvik.test
    • DynamicDalvikComparisonTestForAndroidLibs
  • com.ibm.wala.ide.jdt.test
    • ECJJavaIRTest
    • ECJSyncDuplicatorTest
    • JDTJava15IRTests
    • JDTJavaIRTests
  • com.ibm.wala.ide.jsdt.tests
    • JSProjectScopeTest

I believe that many of these failures are due to resource path / class loader problems. WALA tests commonly need to find other files within the WALA source tree, and use Java class loader facilities to do that. In order for that process to work, Gradle needs to copy the appropriate files into the appropriate run-time resource directories, Gradle needs to set run-time class paths to include those directories, and the test code itself needs to use class loaders that use those class paths.

For the test suites that pass, I have already successfully gotten the class loaders to find the right resources. For the test suites that fail, I am apparently not yet doing this correctly. Some of the testing infrastructure uses class loaders in complex ways, so here I will probably need help from more-experienced WALA maintainers.
@liblit liblit added this to the Gradle and Buildship milestone Dec 29, 2017
@liblit liblit mentioned this issue Jan 11, 2018
Closed
liblit added a commit that referenced this issue Jan 17, 2018
@liblit
Explicitly disable regression tests that are currently failing
f1d2583 
<#5> notes that several
subprojects' tests are currently broken under Gradle.  I'd still like
to be able to run non-broken tests, though.  So here I'm disabling the
failing tests.  The intent is to treat these exclusions as a to-do
list.  We can remove exclusions as we get the corresponding tests
working.  No more exclusions means
<#5> is fixed.
liblit added a commit that referenced this issue Jan 17, 2018
@liblit
Turn on proper regression testing in Travis-CI
f2c50f7 
Many tests are excluded until
<#5> is fixed.  But we can at
least have Travis-CI watching over our shoulder to ensure that
no *new* regressions sneak into the tree.
liblit added a commit that referenced this issue Jan 17, 2018
@liblit
Explicitly disable regression tests that are currently failing
aca5f9b 
<#5> notes that several
subprojects' tests are currently broken under Gradle.  I'd still like
to be able to run non-broken tests, though.  So here I'm disabling the
failing tests.  The intent is to treat these exclusions as a to-do
list.  We can remove exclusions as we get the corresponding tests
working.  No more exclusions means
<#5> is fixed.
liblit added a commit that referenced this issue Jan 17, 2018
@liblit
Turn on proper regression testing in Travis-CI
6ca3213 
Many tests are excluded until
<#5> is fixed.  But we can at
least have Travis-CI watching over our shoulder to ensure that
no *new* regressions sneak into the tree.
liblit added a commit that referenced this issue Jan 17, 2018
@liblit
Explicitly disable regression tests that are currently failing
8992912 
<#5> notes that several
subprojects' tests are currently broken under Gradle.  I'd still like
to be able to run non-broken tests, though.  So here I'm disabling the
failing tests.  The intent is to treat these exclusions as a to-do
list.  We can remove exclusions as we get the corresponding tests
working.  No more exclusions means
<#5> is fixed.
liblit added a commit that referenced this issue Jan 17, 2018
@liblit
Turn on proper regression testing in Travis-CI
02b7ba8 
Many tests are excluded until
<#5> is fixed.  But we can at
least have Travis-CI watching over our shoulder to ensure that
no *new* regressions sneak into the tree.
liblit added a commit that referenced this issue Jan 17, 2018
@liblit
Explicitly disable regression tests that are currently failing
2ad4d71 
<#5> notes that several
subprojects' tests are currently broken under Gradle.  I'd still like
to be able to run non-broken tests, though.  So here I'm disabling the
failing tests.  The intent is to treat these exclusions as a to-do
list.  We can remove exclusions as we get the corresponding tests
working.  No more exclusions means
<#5> is fixed.
liblit added a commit that referenced this issue Jan 17, 2018
@liblit
Turn on proper regression testing in Travis-CI
dbbb64a 
Many tests are excluded until
<#5> is fixed.  But we can at
least have Travis-CI watching over our shoulder to ensure that
no *new* regressions sneak into the tree.
liblit added a commit that referenced this issue Mar 18, 2018
@liblit
Explicitly disable regression tests that are currently failing
c0203cf 
<#5> notes that several
subprojects' tests are currently broken under Gradle.  I'd still like
to be able to run non-broken tests, though.  So here I'm disabling the
failing tests.  The intent is to treat these exclusions as a to-do
list.  We can remove exclusions as we get the corresponding tests
working.  No more exclusions means
<#5> is fixed.
liblit added a commit that referenced this issue Mar 18, 2018
@liblit
Turn on proper regression testing in Travis-CI
2f26bda 
Many tests are excluded until
<#5> is fixed.  But we can at
least have Travis-CI watching over our shoulder to ensure that
no *new* regressions sneak into the tree.
@liblit liblit mentioned this issue Mar 22, 2018
Merged
liblit added a commit that referenced this issue Apr 17, 2018
@liblit
Explicitly disable regression tests that are currently failing
e2e340b 
<#5> notes that several
subprojects' tests are currently broken under Gradle.  I'd still like
to be able to run non-broken tests, though.  So here I'm disabling the
failing tests.  The intent is to treat these exclusions as a to-do
list.  We can remove exclusions as we get the corresponding tests
working.  No more exclusions means
<#5> is fixed.
liblit added a commit that referenced this issue Apr 17, 2018
@liblit
Turn on proper regression testing in Travis-CI
bde936e 
Many tests are excluded until
<#5> is fixed.  But we can at
least have Travis-CI watching over our shoulder to ensure that
no *new* regressions sneak into the tree.
liblit added a commit that referenced this issue Apr 18, 2018
@liblit
Explicitly disable regression tests that are currently failing
5c30dd7 
<#5> notes that several
subprojects' tests are currently broken under Gradle.  I'd still like
to be able to run non-broken tests, though.  So here I'm disabling the
failing tests.  The intent is to treat these exclusions as a to-do
list.  We can remove exclusions as we get the corresponding tests
working.  No more exclusions means
<#5> is fixed.
liblit added a commit that referenced this issue Apr 18, 2018
@liblit
Turn on proper regression testing in Travis-CI
5040019 
Many tests are excluded until
<#5> is fixed.  But we can at
least have Travis-CI watching over our shoulder to ensure that
no *new* regressions sneak into the tree.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug help wanted
Projects
None yet
Milestone
Gradle and Buildship
Development

No branches or pull requests
1 participant
@liblit