New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ow karate #3956
Ow karate #3956
Conversation
there are files that I think should be git ignore, like |
tests/functional/karate_tests/src/test/java/com/karate/openwhisk/utils/sleep.feature
Outdated
Show resolved
Hide resolved
tests/functional/karate_tests/src/test/java/com/karate/openwhisk/wskadmin/WskAdminRunner.java
Outdated
Show resolved
Hide resolved
tests/functional/karate_tests/src/test/java/com/karate/openwhisk/wskadmin/WskAdminRunner.java
Outdated
Show resolved
Hide resolved
You can drop the gradle wrapper related files as they are already part of core repo. So files like graddle-wrapper.jar, gradle.properties, gradlew.bat etc can be removed. Further add this module in settings.gradle in base directory. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can cut down on some boiler plat code to reduce the code further and thus avoid the noise
...s/functional/karate_tests/src/test/java/com/karate/openwhisk/smoketests/SmokeTestRunner.java
Outdated
Show resolved
Hide resolved
...s/functional/karate_tests/src/test/java/com/karate/openwhisk/smoketests/SmokeTestRunner.java
Outdated
Show resolved
Hide resolved
...tests/src/test/java/com/karate/openwhisk/smoketests/TC01_Create-Invoke-Delete-Action.feature
Outdated
Show resolved
Hide resolved
Given Karate guidance on naming convention it would be better to change package name to just |
What's the plan for this PR - is there support to accept it and maintain it? |
We are still using the Karate based tests and maintaining this repo-->(https://github.com/rahulqelfo/ow-karate). |
Codecov Report
@@ Coverage Diff @@
## master #3956 +/- ##
===========================================
+ Coverage 59.35% 80.97% +21.61%
===========================================
Files 162 163 +1
Lines 7585 7594 +9
Branches 500 502 +2
===========================================
+ Hits 4502 6149 +1647
+ Misses 3083 1445 -1638
Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I went through this PR. Can we avoid some some of the duplication - like redundant gradle, some test artifacts, and I think we have a fileutil writer somewhere as well.
My concern with this PR is that it raises the question of where does one add tests when adding new functionality - existing scala tests or karate tests? Do we really get 20% extra coverage from these tests -- I thought we were in the 80s before.
|
||
### How to add a new test type | ||
1. Create a package in `src/test/java`. | ||
2. We can create test types like regression, sanity etc.For example `org.apache.openwhisk.sanitytests` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
space after etc.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Resolved this in the latest commit.
|
||
|
||
### Config Variable(Optional) to run the tests | ||
By default the tests would run using guest namespace.The variables in karate.config |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
using the guest
namespace.
space before The.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Resolved this in the latest commit.
3. Create a feature and runner file inside the above package | ||
|
||
|
||
### Config Variable(Optional) to run the tests |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
space before (Optional)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Resolved this in the latest commit.
@@ -0,0 +1,8 @@ | |||
# Licensed to the Apache Software Foundation (ASF) under one or more contributor |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why do we need another gradle?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Resolved this in the latest commit.
@@ -0,0 +1,45 @@ | |||
function() { | |||
// Licensed to the Apache Software Foundation (ASF) under one or more contributor |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
license should come at the top of the file.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is not supported in Karate. The file must start with function
.
|
||
if (!adminauth) { | ||
|
||
adminauth='d2hpc2tfYWRtaW46c29tZV9wYXNzdzByZA==' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should we hardcode anything in here?
might be better to just abort.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Resolved this in the latest commit.
@@ -0,0 +1,8 @@ | |||
function() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can't reuse existing test artifacts?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Due to the limitation as mentioned above, the existing artifact can't be re-used .Also the project is designed to be completely isolated from the existing ScalaTest and OW code. The idea is to have some sort of distinction from the existing tests
The idea of this project is to introduce another dimension of testing(User Behaviors/Paths) which would complement the existing ScalaTests. So, the ScalaTests would serve more of a Smoke/Sanity/Integrations Tests at the system level and Karate tests would capture user based scenarios. |
@rahulqelfo Do you want to shepherd this PR further? Or see if Manoj will take it up? Otherwise, let's close for now, and reopen later if a new owner emerges. WDYT? |
@tysonnorris The PR has already been updated based on the responses from the reviewers. The only pending item is a decision to merge this and keep Karate Tests within the OW code base. I can look at the jenkins CI failure, once someone can review this PR and suggest if further changes are required. Please suggest. |
Configuration config = new Configuration(new File("target"), "openwhisk"); | ||
ReportBuilder reportBuilder = new ReportBuilder(jsonPaths, config); | ||
reportBuilder.generateReports(); | ||
String dir = System.getProperty("user.dir"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looks like this line and the next are not needed.
""" | ||
if(responseStatusCode==200){ | ||
karate.log("Action got deleted"); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
indentation minus 3? (here and few other places in this file)
@rahulqelfo @tysonnorris I have no strong opinions either way on this PR as long as there's a commitment to maintain these tests as part of the repo. It would be useful to articulate when new tests should be added to this suite and if there's a long term vision for more coverage. |
@rahulqelfo Please chime in on your plans to maintain and extend these tests, otherwise I think we should close this PR. At this point I don't think anyone at Adobe will maintain them, so I think this is up to you. I will verify with Manoj as well, in case he wants to take this up. |
@tysonnorris , I agree that we can close this for now, as I would not be able to devote time on this PR at this moment. We can open it when we need it or find any owner for it. |
Closing per comment on April 18 above. |
This PR consists of functional test cases using Karate BDD Framework.
Description
This PR intends to complement the existing functional tests cases using the Karate BDD framework. The idea of implementing this framework is to enable testers/QA to create complete E2E scenarios utilizing the common testing framework. Also as this Framework supports Gatling, it enables to create Smoke tests based on performance.
Related issue and scope
In process of creating brief wiki explaining about this approach in here-->https://cwiki.apache.org/confluence/display/OPENWHISK/OpenWhisk+Project+Wiki
My changes affect the following components
Types of changes
Checklist: