Remove runtime tests, simplify Rest vs CLI test hierarchy, add sniff test for all runtimes #3840

rabbah · 2018-07-04T04:41:48Z

Additional cleanup of tests:

removes WskRest* where possible and makes the base test suite the Rest tests
removes runtime tests should should be moved to their respective runtimes
runs the unicode tests over all the non-deprecated kinds per the runtime manifest

Also fixes #3845. See issue for details.

Description

Related issue and scope

I opened an issue to propose and discuss this change (tests package needs some cleanup #3737, WskRestOperation does not properly set the exec kind even when one is specified #3845)

My changes affect the following components

Types of changes

Bug fix (generally a non-breaking change which closes an issue).
Enhancement or new feature (adds new functionality).
Breaking change (a bug fix or enhancement which changes existing behavior).

Checklist:

I signed an Apache CLA.
I reviewed the style guides and followed the recommendations (Travis CI will check :).
I added tests to cover my changes.
My changes require further changes to the documentation.
I updated the documentation where necessary.

chetanmeh · 2018-07-04T04:43:24Z

@rabbah Do the unicode test pass for you on Mac?. For me they always failed and only passed on travis. May be related to default encoding and we may need to make it more explicit

rabbah · 2018-07-04T04:44:58Z

They do yes - do you run the tests from gradle or ij?

rabbah · 2018-07-04T04:45:55Z

I just need a sniff tests - it doesn't have to be unicode (which should be covered by the unit proxy tests anyway). So it could be "echo" as well.

rabbah · 2018-07-06T02:00:48Z

tests/src/test/scala/system/basic/WskBasicJavaTests.scala

-  /**
-   * Test the Java "hello world" demo sequence
-   */
-  it should "Invoke a java action" in withAssetCleaner(wskprops) { (wp, assetHelper) =>


this test can be dropped.

rabbah · 2018-07-06T02:01:07Z

tests/src/test/scala/system/basic/WskBasicJavaTests.scala

-  /*
-   * Example from the docs.
-   */
-  it should "Invoke a Java action where main is in the default package" in withAssetCleaner(wskprops) {


move this to unit test.

EDIT: added to java runtime tests.

rabbah · 2018-07-06T02:01:14Z

tests/src/test/scala/system/basic/WskBasicJavaTests.scala

-      }
-  }
-
-  it should "Ensure that Java actions cannot be created without a specified main method" in withAssetCleaner(wskprops) {


move this to cli/unit test.

EDIT: moved.

rabbah · 2018-07-06T02:02:18Z

tests/src/test/scala/system/basic/WskBasicNode6Tests.scala

-
-  behavior of s"Runtime $currentNodeJsKind"
-
-  it should "Ensure that NodeJS actions can have a non-default entrypoint" in withAssetCleaner(wskprops) {


drop, duplicate of unit test.

rabbah · 2018-07-06T02:03:17Z

tests/src/test/scala/system/basic/WskBasicNode6Tests.scala

-      }
-  }
-
-  it should "Ensure that zipped actions are encoded and uploaded as NodeJS actions" in withAssetCleaner(wskprops) {


drop, duplicate of unit test.

EDIT: moved to CLI test suite.

rabbah · 2018-07-06T02:08:16Z

tests/src/test/scala/system/basic/WskBasicPythonTests.scala

-    }
-  }
-
-  it should "invoke an action and confirm expected environment is defined" in withAssetCleaner(wskprops) {


drop, duplicate.

rabbah · 2018-07-06T02:09:18Z

tests/src/test/scala/system/basic/WskBasicPythonTests.scala

-      }
-  }
-
-  it should "invoke an invalid action and get error back" in withAssetCleaner(wskprops) { (wp, assetHelper) =>


duplicate - but unit test doesn't check error message as done here, should tighten unit test accordingly.

EDIT: updated unit tests for runtimes accordingly.

rabbah · 2018-07-06T02:10:07Z

tests/src/test/scala/system/basic/WskBasicPythonTests.scala

-
-  behavior of "Python virtualenv"
-
-  Seq(("python2_virtualenv.zip", "python:2"), ("python3_virtualenv.zip", "python:3")).foreach {


drop, duplicates.

rabbah · 2018-07-06T02:10:45Z

tests/src/test/scala/system/basic/WskBasicSwift3Tests.scala

-
-  behavior of "Swift runtime"
-
-  it should "Ensure that Swift actions can have a non-default entrypoint" in withAssetCleaner(wskprops) {


drop, duplicate.

rabbah · 2018-07-06T02:14:52Z

tests/src/test/scala/system/basic/WskRestActionTests.scala

-class WskRestActionTests extends WskActionTests with WskActorSystem {
-  override val wsk = new WskRestOperations
-
-  it should "create an action with an empty file" in withAssetCleaner(wskprops) { (wp, assetHelper) =>


The docker skeleton and python runtimes accept an empty code string during init, and fail in run (docker skeleton will just run the stub, and python will return error = action did not return dictionary).

Java will fail with class/method not found for main during class loader.

Swift will fail with did not generate an executable.

Dropping this test is OK - I don't think we're getting any increased coverage. It would be nice to make the failures uniform but this may be a little difficult for all the runtimes to normalize.

See #3839 and #3844.

EDIT: I added tests for empty code strings to the runtime suite.

rabbah · 2018-07-06T19:22:23Z

tests/src/test/scala/common/rest/WskRestOperations.scala

-
-    val bodyHead = Map("name" -> name.toJson, "namespace" -> namespace.toJson)
-
-    val (updateExistence, resultExistence) = update match {


I removed this because this is mimicking CLI functionality and I think should be considered invalid for the REST operations. If this was an SDK I can see supporting this but here the tests should be making pass through calls and not really doing anything fancy. I kept copy support although I was tempted to remove that also.

rabbah · 2018-07-06T19:24:16Z

tests/src/test/scala/whisk/core/cli/test/WskRestBasicUsageTests.scala

    }
  }

-  it should "reject action update for sequence with no components" in withAssetCleaner(wskprops) { (wp, assetHelper) =>


moved to a unit test.

rabbah · 2018-07-06T19:24:26Z

tests/src/test/scala/whisk/core/cli/test/WskRestBasicUsageTests.scala

      }
  }

-  it should "report error when creating an action with unknown kind" in withAssetCleaner(wskprops) {


moved to a unit test.

rabbah · 2018-07-06T19:24:32Z

tests/src/test/scala/whisk/core/cli/test/WskRestBasicUsageTests.scala

-      rr.stderr should include("kind 'foobar' not in Set")
-  }
-
-  it should "report error when creating an action with zip but without kind" in withAssetCleaner(wskprops) {


moved to a unit test.

rabbah · 2018-07-06T19:24:48Z

tests/src/test/scala/whisk/core/cli/test/WskRestBasicUsageTests.scala

    }
  }

-  it should "ensure action update with --web flag only copies existing annotations when new annotations are not provided" in withAssetCleaner(


this is a CLI test, does not belong with the REST tests.

EDIT: already exists in the cli tests.

codecov-io · 2018-07-06T20:10:40Z

Codecov Report

Merging #3840 into master will increase coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #3840      +/-   ##
==========================================
+ Coverage    74.6%   74.61%   +0.01%     
==========================================
  Files         138      138              
  Lines        6465     6465              
  Branches      406      406              
==========================================
+ Hits         4823     4824       +1     
+ Misses       1642     1641       -1

Impacted Files	Coverage Δ
.../scala/src/main/scala/whisk/core/entity/Exec.scala	`83.76% <0%> (+0.64%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9c05de4...d2926f7. Read the comment docs.

dgrove-oss · 2018-07-06T20:20:38Z

PG1 3103 🔴

…ect repo.

Add canonical unicode tests.

…T test.

markusthoemmes · 2018-07-08T13:36:30Z

Thanks a big bunch for all of this grunt work 👍

Tweak some tests checking status codes. Fix bug in test which permitted one too many log sentinels. Add test for non-main entry point Add one more tests to runtime docs.

See apache/openwhisk#3840.

rabbah · 2018-07-09T02:28:17Z

@dgrove-oss All the tests which needed to move have no been moved. This PR is ready. Linked all related PRs:

apache/openwhisk-cli#350
apache/openwhisk-runtime-docker#52 (and git tag as 1.3.2)
apache/openwhisk-runtime-swift#72 (and git tag)
apache/openwhisk-runtime-python#33 (and git tag)
apache/openwhisk-runtime-nodejs#71 (and git tag)
apache/openwhisk-runtime-java#63 (and git tag)
apache/openwhisk-runtime-php#35 (and git tag)

dgrove-oss

looks good to go. I will merge this and then work on merging through all the downstream PRs.

* Update tests for upstream changes in apache/openwhisk#3840.

…test for all runtimes (apache#3840) Additional cleanup of tests: removes WskRest* where possible and makes the base test suite the Rest tests removes runtime tests should should be moved to their respective runtimes runs the unicode tests over all the non-deprecated kinds per the runtime manifest

rabbah mentioned this pull request Jul 5, 2018

Add test for empty initializer. Document other tests. #3844

Merged

21 tasks

rabbah force-pushed the tests branch from d28bf2e to 4870c99 Compare July 6, 2018 02:27

rabbah commented Jul 6, 2018

View reviewed changes

rabbah force-pushed the tests branch 2 times, most recently from ee8be72 to 7c660ae Compare July 6, 2018 03:06

rabbah mentioned this pull request Jul 6, 2018

extend action proxy tests to increase uniformity between action runtimes #3839

Closed

4 tasks

rabbah force-pushed the tests branch from 7c660ae to 761b78e Compare July 6, 2018 04:24

rabbah added testing review Review for this PR has been requested and yet needs to be done. labels Jul 6, 2018

rabbah force-pushed the tests branch 2 times, most recently from 0968438 to d38af10 Compare July 6, 2018 18:13

rabbah mentioned this pull request Jul 6, 2018

Allow explicitly specified kind to take precedence. #3847

Closed

21 tasks

rabbah force-pushed the tests branch from ed8b5c6 to d9e8f33 Compare July 6, 2018 19:25

rabbah commented Jul 6, 2018

View reviewed changes

rabbah force-pushed the tests branch from d9e8f33 to f3ab93e Compare July 6, 2018 19:26

rabbah assigned dgrove-oss Jul 6, 2018

rabbah requested a review from dgrove-oss July 6, 2018 19:27

rabbah force-pushed the tests branch 5 times, most recently from a6c27dc to a71efae Compare July 7, 2018 16:22

rabbah added 3 commits July 7, 2018 12:56

Remove runtime/language tests; will move relevant tests to their resp…

418c911

…ect repo.

Run unicode tests based on runtime manifest from API host.

643122b

Add canonical unicode tests.

Fix bugs in wsk.action.create for REST bindings.

31ecf4d

rabbah added 2 commits July 7, 2018 12:56

Fix bug in newly enabled test.

39bd17b

Move some tests to units test, and remove test that's not a valid RES…

b6390bd

…T test.

rabbah force-pushed the tests branch 3 times, most recently from 7ac0094 to 9e470e3 Compare July 8, 2018 01:01

rabbah mentioned this pull request Jul 8, 2018

Simplify interface for canonical runtime tests suite #3851

Closed

22 tasks

rabbah force-pushed the tests branch from 9e470e3 to e128323 Compare July 9, 2018 01:39

Simplify test interface for runtimes.

a2448ce

Tweak some tests checking status codes. Fix bug in test which permitted one too many log sentinels. Add test for non-main entry point Add one more tests to runtime docs.

rabbah force-pushed the tests branch from e128323 to a2448ce Compare July 9, 2018 02:03

rabbah added a commit to rabbah/incubator-openwhisk-cli that referenced this pull request Jul 9, 2018

Update tests for upstream changes.

c049287

See apache/openwhisk#3840.

rabbah mentioned this pull request Jul 9, 2018

Update tests for upstream changes. apache/openwhisk-cli#350

Merged

Remove unused file.

d2926f7

This was referenced Jul 9, 2018

Don't check for sentinels in reinit test #3854

Closed

Do not write sentinels on a second init apache/openwhisk-runtime-php#36

Closed

dgrove-oss approved these changes Jul 9, 2018

View reviewed changes

dgrove-oss merged commit 5b13c4c into apache:master Jul 9, 2018

dgrove-oss pushed a commit to apache/openwhisk-cli that referenced this pull request Jul 9, 2018

Update tests for upstream changes. (#350)

73331c6

* Update tests for upstream changes in apache/openwhisk#3840.


		behavior of s"Runtime $currentNodeJsKind"

		it should "Ensure that NodeJS actions can have a non-default entrypoint" in withAssetCleaner(wskprops) {


		behavior of "Python virtualenv"

		Seq(("python2_virtualenv.zip", "python:2"), ("python3_virtualenv.zip", "python:3")).foreach {


		behavior of "Swift runtime"

		it should "Ensure that Swift actions can have a non-default entrypoint" in withAssetCleaner(wskprops) {


		val bodyHead = Map("name" -> name.toJson, "namespace" -> namespace.toJson)

		val (updateExistence, resultExistence) = update match {

Remove runtime tests, simplify Rest vs CLI test hierarchy, add sniff test for all runtimes #3840

Remove runtime tests, simplify Rest vs CLI test hierarchy, add sniff test for all runtimes #3840

Uh oh!

Conversation

rabbah commented Jul 4, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related issue and scope

My changes affect the following components

Types of changes

Checklist:

Uh oh!

chetanmeh commented Jul 4, 2018

Uh oh!

rabbah commented Jul 4, 2018

Uh oh!

rabbah commented Jul 4, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rabbah Jul 6, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rabbah Jul 6, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rabbah Jul 6, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rabbah Jul 6, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rabbah Jul 6, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rabbah Jul 6, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov-io commented Jul 6, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

dgrove-oss commented Jul 6, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

markusthoemmes commented Jul 8, 2018

Uh oh!

rabbah commented Jul 9, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dgrove-oss left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

rabbah commented Jul 4, 2018 •

edited

Loading

rabbah Jul 6, 2018 •

edited

Loading

rabbah Jul 6, 2018 •

edited

Loading

rabbah Jul 6, 2018 •

edited

Loading

rabbah Jul 6, 2018 •

edited

Loading

rabbah Jul 6, 2018 •

edited

Loading

rabbah Jul 6, 2018 •

edited

Loading

codecov-io commented Jul 6, 2018 •

edited

Loading

dgrove-oss commented Jul 6, 2018 •

edited

Loading

rabbah commented Jul 9, 2018 •

edited

Loading