[Backport 2.x] Integration tests framework #3388

peternied · 2023-09-21T21:16:35Z

Description

Test framework that allows for declaring within the test class all
related users, roles, and other security primitives. Emphasis on
hamcrest Matchers for straight forward to read, author, and
troubleshoot test cases.

Special Note

This is a giant pull request - I would highly recommend using the github.dev [view] for reviewing and providing feedback. I'll also provide comments for the initial reviewers where I call out any deviations I've taken from what is in main

Issues Resolved

Resolves Backport the test framework to 2.x branch #3230

Check List

New functionality includes testing
New functionality has been documented
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Test framework that allows for declaring within the test class all related users, roles, and other security primatives. Empahsis on hamcrest assertMatchers for straight forward to read, author, and troubleshoot test cases. Signed-off-by: Peter Nied <petern@amazon.com>

peternied · 2023-09-21T21:24:26Z

Signed-off-by: Peter Nied <petern@amazon.com>

codecov · 2023-09-21T21:48:01Z

Codecov Report

Merging #3388 (ba3a701) into 2.x (a2daf9f) will increase coverage by 0.16%.
Report is 7 commits behind head on 2.x.
The diff coverage is n/a.

@@             Coverage Diff              @@
##                2.x    #3388      +/-   ##
============================================
+ Coverage     64.11%   64.28%   +0.16%     
- Complexity     3367     3458      +91     
============================================
  Files           256      256              
  Lines         19515    19515              
  Branches       3297     3297              
============================================
+ Hits          12512    12545      +33     
+ Misses         5360     5355       -5     
+ Partials       1643     1615      -28

Files	Coverage Δ
...curity/dlic/rest/validation/EndpointValidator.java	`94.20% <ø> (ø)`

... and 15 files with indirect coverage changes

Signed-off-by: Peter Nied <petern@amazon.com>

.github/workflows/ci.yml

build.gradle

Signed-off-by: Peter Nied <petern@amazon.com>

peternied · 2023-09-22T01:55:29Z

Note; Moving this to draft while I'm making sure these tests are stable, I've added a bunch of debug only changes that should be reverted before merging, I'll pull this out of draft when its mergeable - but feel free to review it as it stands

Signed-off-by: Peter Nied <petern@amazon.com>

peternied · 2023-09-22T20:43:13Z

Ok here is the actual numbers that are telling:

$ ls | grep "integration.*txt" | grep ubuntu | xargs -I {} tail -n 300 {} | grep "\- org" | awk -F' - ' '{print $2}' | sort | uniq -c
      1 org.opensearch.security.PointInTimeOperationTest.listAllPits_positive
      1 org.opensearch.security.SearchOperationTest.shouldCreateIndexTemplate_positive
      2 org.opensearch.security.SearchOperationTest.shouldDeleteDocumentInBulk_partiallyPositive
      3 org.opensearch.security.SearchOperationTest.shouldIndexDocumentInBulkRequest_partiallyPositive
      2 org.opensearch.security.SearchOperationTest.shouldReindexDocuments_positive
      1 org.opensearch.security.SearchOperationTest.shouldRestoreSnapshot_failureForbiddenIndex
      1 org.opensearch.security.SearchOperationTest.shouldRestoreSnapshot_failureOperationForbidden
      4 org.opensearch.security.SearchOperationTest.shouldRestoreSnapshot_positive
      1 org.opensearch.security.SearchOperationTest.shouldUpdateDocumentsInBulk_partiallyPositive
      2 org.opensearch.security.SearchOperationTest.shouldUpdateDocumentsInBulk_positive
      3 org.opensearch.security.SearchOperationTest.shouldUpdateTemplate_positive

linux failures ^

$ ls | grep "integration.*txt" | grep win | xargs -I {} tail -n 300 {} | grep "\- org" | awk -F' - ' '{print $2}' | sort | uniq -c
     12 org.opensearch.security.DlsIntegrationTests.classMethod
      2 org.opensearch.security.DlsIntegrationTests.testSearchForAllDocumentsWithIndexPattern
      3 org.opensearch.security.DlsIntegrationTests.testShouldSearchAll
     12 org.opensearch.security.DoNotFailOnForbiddenTests.classMethod
     12 org.opensearch.security.FlsAndFieldMaskingTests.classMethod
     12 org.opensearch.security.PointInTimeOperationTest.classMethod
     12 org.opensearch.security.SearchOperationTest.classMethod
      1 org.opensearch.security.SearchOperationTest.shouldDeleteDocumentInBulk_positive
      1 org.opensearch.security.SearchOperationTest.shouldIndexDocumentInBulkRequest_partiallyPositive
     12 org.opensearch.security.SecurityConfigurationTests.classMethod
     12 org.opensearch.security.SecurityRolesTests.classMethod
     12 org.opensearch.security.SslOnlyTests.classMethod
     12 org.opensearch.security.TlsTests.classMethod
     12 org.opensearch.security.api.CreateResetPasswordTest.classMethod
     12 org.opensearch.security.api.DashboardsInfoTest.classMethod
     12 org.opensearch.security.api.DashboardsInfoWithSettingsTest.classMethod
     12 org.opensearch.security.http.CertificateAuthenticationTest.classMethod
     12 org.opensearch.security.privileges.PrivilegesEvaluatorTest.classMethod

windows failures ^

Signed-off-by: Peter Nied <petern@amazon.com>

peternied · 2023-09-25T22:12:23Z

OK - I'm pretty sure this is ready to merge, or near as ready as I'm going to try to push it, I've made a number of annotations to ignore flaky tests, we should revisit these and I'll add several tracking issues, along with disabling windows due to the network trouble.

Proof that this test collection is stable - 10x runs on JDK 11 / 17

https://github.com/opensearch-project/security/actions/runs/6305277168/job/17118384937?pr=3388

Signed-off-by: Peter Nied <peternied@hotmail.com>

willyborankin

LGTM

peternied · 2023-09-27T11:59:30Z

@RyanL1997 @DarshitChanpura could I get another review?

DarshitChanpura

LGTM. I left one question out of curiosity.

Nit: There are some new files being introduced with floraGunn license headers (there is one more license header as well in a couple of files). We should clean this up at some point. Not necessarily in this PR, as these need to be fixed in main as well.

build.gradle

RyanL1997

Hi @peternied, thanks for taking this on and in generally this is looking good to me. I just left a minor comment for the httpcomponents dependencies.

build.gradle

peternied · 2023-09-29T20:04:01Z

@DarshitChanpura & @RyanL1997 see [1] for the fast follow up items.

[1] Minor comments follow up for integration tests #3427

Test framework that allows for declaring within the test class all related users, roles, and other security primitives. Emphasis on hamcrest Matchers for straight forward to read, author, and troubleshoot test cases. Signed-off-by: Peter Nied <petern@amazon.com> Signed-off-by: Peter Nied <peternied@hotmail.com> Signed-off-by: Peter Nied <petern@amazon.com>

- Remove an unused JVM setting for integration tests - Fixed an issue where you couldn't run non-resource tests unless you had CI_ENVIRONMENT set to normal on your machine (Already fixed in main) - Fixed an issue where code coverage was always run after tests completed (Already fixed in main) - Related #3388 Signed-off-by: Peter Nied <petern@amazon.com>

- Remove an unused JVM setting for integration tests - Fixed an issue where you couldn't run non-resource tests unless you had CI_ENVIRONMENT set to normal on your machine (Already fixed in main) - Fixed an issue where code coverage was always run after tests completed (Already fixed in main) - Related opensearch-project#3388 Signed-off-by: Peter Nied <petern@amazon.com>

peternied requested review from cliu123, cwperks, DarshitChanpura, davidlago, RyanL1997, stephen-crawford, reta and willyborankin as code owners September 21, 2023 21:16

peternied added 3 commits September 21, 2023 21:38

Remove OBO config references

c8bed13

Signed-off-by: Peter Nied <petern@amazon.com>

Add GetWithBody based off of HttpPost

9c50026

Signed-off-by: Peter Nied <petern@amazon.com>

Fix password valiation message

cd6159b

Signed-off-by: Peter Nied <petern@amazon.com>

peternied added 4 commits September 21, 2023 21:54

Fix Dashboard info tests

e68a6d6

Signed-off-by: Peter Nied <petern@amazon.com>

Remove tests for features that aren't in 2.x

ccd3239

Signed-off-by: Peter Nied <petern@amazon.com>

Fix missing header

ca93497

Signed-off-by: Peter Nied <petern@amazon.com>

Fix intermittant issue with number of requests

ef6c383

Signed-off-by: Peter Nied <petern@amazon.com>

peternied commented Sep 21, 2023

View reviewed changes

.github/workflows/ci.yml Show resolved Hide resolved

peternied commented Sep 21, 2023

View reviewed changes

.github/workflows/ci.yml Show resolved Hide resolved

peternied commented Sep 21, 2023

View reviewed changes

build.gradle Show resolved Hide resolved

peternied added 4 commits September 21, 2023 22:49

Synchronize getWithJsonBody correctly

12f170b

Signed-off-by: Peter Nied <petern@amazon.com>

Fix linngering CI issues + License headers

7a1296d

Signed-off-by: Peter Nied <petern@amazon.com>

Make sure checkstyle also runs on integration tests

01c48d1

Signed-off-by: Peter Nied <petern@amazon.com>

Slim down log output and amp up number of tests for troubleshooting

5f9002f

Signed-off-by: Peter Nied <petern@amazon.com>

peternied marked this pull request as draft September 22, 2023 01:54

peternied added 3 commits September 22, 2023 01:59

Revert log4j changes, to prevent test failurse

e94d8c1

Signed-off-by: Peter Nied <petern@amazon.com>

Ignore Cross Cluster Tests

52e6c6e

Signed-off-by: Peter Nied <petern@amazon.com>

Disable another cross cluster test

c2af087

Signed-off-by: Peter Nied <petern@amazon.com>

peternied added 4 commits September 25, 2023 21:53

Just disabling tests

907fe4f

Signed-off-by: Peter Nied <petern@amazon.com>

Add brief descriptions on why integration tests are marked ignored

14530d3

Signed-off-by: Peter Nied <petern@amazon.com>

Remove ignores too far

f3cef35

Signed-off-by: Peter Nied <petern@amazon.com>

Update CI workflow for these tests

a714cf5

Signed-off-by: Peter Nied <petern@amazon.com>

peternied marked this pull request as ready for review September 25, 2023 22:10

Fix spotless issue on ci.yml

ba3a701

Signed-off-by: Peter Nied <peternied@hotmail.com>

willyborankin approved these changes Sep 27, 2023

View reviewed changes

DarshitChanpura approved these changes Sep 27, 2023

View reviewed changes

build.gradle Show resolved Hide resolved

stephen-crawford approved these changes Sep 27, 2023

View reviewed changes

RyanL1997 approved these changes Sep 27, 2023

View reviewed changes

build.gradle Show resolved Hide resolved

peternied merged commit 19c6a9d into opensearch-project:2.x Sep 29, 2023
56 checks passed

peternied deleted the 2.x-intergration-tests branch September 29, 2023 19:00

This was referenced Sep 29, 2023

[Tests] integration-tests are not being run on windows #3423

Closed

Minor comments follow up for integration tests #3427

Merged

peternied mentioned this pull request Oct 10, 2023

Forward port integration tests #3512

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Backport 2.x] Integration tests framework #3388

[Backport 2.x] Integration tests framework #3388

peternied commented Sep 21, 2023 •

edited

Loading

peternied commented Sep 21, 2023 •

edited

Loading

codecov bot commented Sep 21, 2023 •

edited

Loading

peternied commented Sep 22, 2023

peternied commented Sep 22, 2023 •

edited

Loading

peternied commented Sep 25, 2023

willyborankin left a comment

peternied commented Sep 27, 2023

DarshitChanpura left a comment

RyanL1997 left a comment

peternied commented Sep 29, 2023

[Backport 2.x] Integration tests framework #3388

[Backport 2.x] Integration tests framework #3388

Conversation

peternied commented Sep 21, 2023 • edited Loading

Description

Special Note

Issues Resolved

Check List

peternied commented Sep 21, 2023 • edited Loading

Resolved, submitting another run

codecov bot commented Sep 21, 2023 • edited Loading

Codecov Report

peternied commented Sep 22, 2023

peternied commented Sep 22, 2023 • edited Loading

peternied commented Sep 25, 2023

willyborankin left a comment

Choose a reason for hiding this comment

peternied commented Sep 27, 2023

DarshitChanpura left a comment

Choose a reason for hiding this comment

RyanL1997 left a comment

Choose a reason for hiding this comment

peternied commented Sep 29, 2023

peternied commented Sep 21, 2023 •

edited

Loading

peternied commented Sep 21, 2023 •

edited

Loading

codecov bot commented Sep 21, 2023 •

edited

Loading

peternied commented Sep 22, 2023 •

edited

Loading