Scalability Test: Can CNF increase and decrease capacity?

[lixuna]

Acceptance Criteria

Scalability Test: Can CNF increase and decrease capacity?

EPIC: CNF Conformance Test Suite

Release: v0.0.1

Test Category: Scalability

Type of test (static or runtime) = Runtime

• example code: lfn-cnti/certification@6a72243

Environment set up tasks:

• Copy admin.conf (the kubeconfig) to new file name $HOME/shared/conformance01.kubeconfig (for accessing the shared conformance cluster)
• Get SSH access to one of the shared dev machines to use as your test workstation
  • Ask @denverwilliams, @wavell, @nupejosh or @taylor
• Talk with @denverwilliams, @wavell to get access to the KUBECONFIG file
• Set the KUBECONFIG environment variable to point to the shared kubeconfig file for accessing the target K8s cluster export KUBECONFIG=~/shared/conformance01.kubeconfig)
• Follow the installation instructions to set up the the conformance suite for development and testing
• Update installation instructions to cover new setup and existing workstation updates. Example:
  • You should have a working conformance suite that compiles and runs correctly. (eg. crystal-lang installed)
  • If you have it installed, pull down latest version of CNF conformance test code from develop branch (eg. git fetch ; get rebase from the cnf-conformance directory)
  • Run shard install to get dependencies

Upstream tool set up tasks: (conformance suite + upstream tools)

• N/A

CNF setup Tasks

• Use the https://github.com/cncf/cnf-conformance/tree/master/cnfs for adding target CNFs to be tested
• Create example conformance configuration file for configuring the use of the target CNF
• Create example environment configuration that has the CNF container name list
• Create code to use the conformance CNF configuration and target folder for testing the CNF
• Add documentation for testing a target CNFs with the cnformance suite (eg. add to usage md). Uncludes
  • Path to Helm chart under cnfs folder
  • Options to pass helm chart
• Add documentation for using target CNFs with the cnformance suite (eg. add to usage md), Include
  • The name of the containers needs to be configured

Sample CNF tasks:

• Create a sample-cnf folder
• Create a subfolder for the sample cnf (eg. dns-cnf1)
• Create a sample CNF using a minimal container which requires no host or vanilla K8s modifications (no extensions) --> eg. CoreDNS container
• Create sample CNF conformance configuration file for using the sample CNF
• Create sample environment configuration that has the sample CNF container name list
• Add TL;DR doc on using the sample in the sample folder

Code implementation tasks:

• If one test blows up (eg. CNF code not found for static) the suite should continue to run
• Add util task which can deploy a CNF that has "valid conformance deploy configuration"
  • helm chart v3.0
  • helm chart path
  • containers name
  • NOTE: does not run when tests run (include all)
• Create new increase test which increases the replica set
  • Check and save current replica count
  • At start set replica to 1 (overriding CNF defaults)
  • Show data from cluster about container
  • Do increase
  • Show data from cluster about container
  • Reset replica back to original saved replica count
  • Show pass or fail for doing increase
  • Show green for pass and red for fail
• Create new decrease test which decreases the repica set
  • Check and save current replica count
  • At start set replica to more than 1 (Eg. 3; overriding CNF defaults)
  • Show data from cluster about container
  • Do decrease
  • Show data from cluster about container
  • Reset replica back to original saved replica count
  • Show pass or fail for doing decrease
  • Show green for pass and red for fail
• Update existing increase_decrease test to call the individual increase and decrease
• Think about and document how we can roll up a summary for sets of tests => #27 updates to add mutli-node cluster as pre-req lfn-cnti/certification#30
• Scalability category should call the increase and decrease test set
• The all test command should run the increase and decrease tests
• Add the increase test to the "runtime test set"
• Add the decrease test to the "runtime test set"
• Optionally, Add the "increase+decrease test set" to the "runtime test set"

Documentation tasks:

• Update Test Categories md if needed
• Update Pseudo Code md if needed
• Update USAGE md if needed
• Update installation instructions if needed

QA tasks

Dev Review:

• walk through A/C
• do you get the expected result?

Steps after A/C Passes for developer:

If all A/C passes for developer

• open a Pull Request to master branch and mark as "developer approved"
• Tag team members for peer review

Peer review:

• walk through A/C
• do you get the expected result?
• if yes, move to Reviewer Approved column
• if no, document what did not go as expected, including error messages and screenshots (if possible)

Steps after A/C fails

If all A/C fails for peer reviewer

• Add a comment to the PR summarizing the problem tagging the developer
• Attach any console output or logs that can help the developer fix the issue

Steps after A/C Passes for peer reviewer

• Mark the PR as approved and merge the PR into master

[lixuna]

Acceptance Criteria for peer review:

---

Common steps

• Find or create a K8s cluster to use
  - [ ] Steps for creating will be in another ticket
  - [ ] Talk with team for access to an existing cluster
• Set the KUBECONFIG environment variable for accessing the target K8s cluster
• Pull down latest version of CNF conformance test code from develop branch
• Run shard install
• Find the names of the containers in the CNF's helm chart
• Create an environment variable called TARGET_CONTAINER with the list of container names

When using the test increase_decrease_capacity command the CNF should PASS the increase and decrease test successfully:

• Run crystal src/cnf-conformance.cr increase_decrease_capacity
• I expect to see the output and results from the increase test
• I expect to see the output and results from the decrease test
• I expect to see PASSED for the increase test
• I expect to see PASSED for the decrease test

The CNF should scale-up in capacity when the increase successfully

• Follow common steps above
• Run command

crystal src/cnf-conformance.cr increase_capacity

• There will be output on the screen showing the current replica set count (eg. 1 container) before the increase happens
  • eg. Test runs kubectl describe command
• Behind the scenes: The test Increase containers to "more than default"
• There will be output on the screen showing the new replicate set count (eg. 3 containers)
  • eg. Test runs kubectl describe command
• Behind the scenes: The test resets the containers to default (Eg. 1)
• There will be output showing if the test PASSED or FAILED
• I would expect the output to say PASSED'

The CNF should scale-down in capacity when the decrease successfully

• Follow common steps above
• Run command

crystal src/cnf-conformance.cr decrease_capacity

• Behind the scenes: The test Increase containers to "default that is more than 1 replica set"
• There will be output on the screen showing the current replica set count (eg. 3 container) before the decrease happens
  • eg. Test runs kubectl describe command
• Behind the scenes: The test decrease containers to "less than default"
• There will be output on the screen showing the new replicate set count (eg. 1 containers)
  • eg. Test runs kubectl describe command
• Behind the scenes: The test resets the containers to a default (Eg. 1 or 3)
• There will be output showing if the test PASSED or FAILED
• I would expect the output to say PASSED'

I would expect the CNF to pass the increase and decrease test when the scalability category runs

• Follow common steps above
• Run crystal src/cnf-conformance.cr scalability
• I expect to see the output and results from the increase and decrease grouped test
• I expect to see PASSED for the increase test
• I expect to see PASSED for the decrease test

I would expect the CNF to pass the increase and decrease test when I run the entire test suite (all)

• Follow common steps above
• Run crystal src/cnf-conformance.cr all
• I expect to see the output from the increase test
• I expect to see the output from the decrease test
• I expect to see PASSED for the increase test
• I expect to see PASSED for the decrease test

---

Steps after A/C fails

If all A/C fails for peer reviewer

• Add a comment to the PR summarizing the problem tagging the developer
• Attach any console output or logs that can help the developer fix the issue

Steps after A/C Passes

If all A/C passes for developer

• open a Pull Request to master branch and mark as "developer approved"
• Tag team members for peer review

If all A/C passes for peer reviewer

• Mark the PR as approved and merge the PR into master

[taylor]

Initial task planning https://hackmd.io/Be2pyIA4Tue-HMT3KdfOPA

[lixuna]

Level of effort in pts: 0, 1, 2, 3, 5, 8, 13, 21, infinity: 8 pts

@wavell 8 pts
@williscool 13 pts
@lixuna 8 pts
@denverwilliams 8 pts
@taylor 8 pts
ashleigh 8 pts
@nupejosh 8 pts

[lixuna]

PoC #22

[lixuna]

@taylor went through A/C and all steps passed, closing issue