Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[cleanup] Deduplicate test certificates to simplify management #20289

Merged

Conversation

michaeljmarshall
Copy link
Member

@michaeljmarshall michaeljmarshall commented May 10, 2023

Motivation

While working on my draft to enable hostname verification by default (#20268), I ran into some issues with certificates and noticed that we have several that are unnecessarily duplicated. This PR removes those duplicates and configures the affected tests to retrieve the certificates from the central location.

Modifications

  • Remove duplicated certificates from several tests
  • Update configurations to get the certificates from the correct location
  • In AuthedAdminProxyHandlerTest, configure proxy to use proxy certs instead of broker certs. This is mostly a cosmetic change because they share the same root CA and they are on the same host, but it seems more correct.

Verifying this change

I verified this by checking that the certs were the same. When tests pass, we can have a high confidence that this change is complete.

Documentation

  • doc-not-needed

Matching PR in forked repository

PR in forked repository: I ran all affected tests locally and got them to pass, so skipping the forked tests.

@michaeljmarshall michaeljmarshall added area/test type/cleanup Code or doc cleanups e.g. remove the outdated documentation or remove the code no longer in use ready-to-test labels May 10, 2023
@michaeljmarshall michaeljmarshall added this to the 3.1.0 milestone May 10, 2023
@michaeljmarshall michaeljmarshall self-assigned this May 10, 2023
@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label May 10, 2023
@codecov-commenter
Copy link

Codecov Report

Merging #20289 (78d5f50) into master (bc1764f) will increase coverage by 38.42%.
The diff coverage is 66.66%.

Impacted file tree graph

@@              Coverage Diff              @@
##             master   #20289       +/-   ##
=============================================
+ Coverage     34.48%   72.90%   +38.42%     
- Complexity    12528    31953    +19425     
=============================================
  Files          1614     1868      +254     
  Lines        126177   138588    +12411     
  Branches      13772    15237     +1465     
=============================================
+ Hits          43508   101037    +57529     
+ Misses        77051    29509    -47542     
- Partials       5618     8042     +2424     
Flag Coverage Δ
inttests 24.18% <38.88%> (+0.05%) ⬆️
systests 24.75% <11.11%> (?)
unittests 72.20% <66.66%> (+39.10%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...r/broker/authentication/AuthenticationService.java 69.52% <37.50%> (+35.52%) ⬆️
...oker/service/nonpersistent/NonPersistentTopic.java 74.29% <66.66%> (+23.86%) ⬆️
...rvice/nonpersistent/NonPersistentSubscription.java 53.29% <100.00%> (+16.04%) ⬆️
...va/org/apache/pulsar/client/impl/ConsumerImpl.java 76.14% <100.00%> (+15.95%) ⬆️
...he/pulsar/client/impl/MultiTopicsConsumerImpl.java 78.26% <100.00%> (+30.98%) ⬆️

... and 1503 files with indirect coverage changes

Copy link
Member

@lhotari lhotari left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lhotari lhotari merged commit 96367e1 into apache:master May 10, 2023
45 checks passed
@michaeljmarshall michaeljmarshall deleted the simplify-cert-management-in-tests branch May 10, 2023 14:53
michaeljmarshall added a commit that referenced this pull request May 17, 2023
Builds on: #20289

### Motivation

There are many certificates in our test code base. It would be much simpler to have one place were we create and manage certificates so that when we need to make changes, they are consolidated.

There is likely one or two more PRs to finish consolidating certs.

### Modifications

* Remove certs that are no longer used
* Replace references to old certs with references to the `certificate-authority` certs
* Create new server certs with valid hostnames on them so that tests will pass. Document the process used to create these certs.
* Fix an issue in the `PulsarTestContext` where the configuration was not correctly updated.
* Remove configurations that allow for insecure connections in tests that are doing some kind of TLS verification. The only places where we leave insecure validation in place is tests that are specifically verifying the functionality.
* Copy `certificate-authority` to the relevant `bouncy-castle` directory

### Verifying this change

When tests pass, this change will be correctly verified.

### Documentation

- [x] `doc`
This PR includes doc changes

### Matching PR in forked repository

PR in forked repository: michaeljmarshall#48
horizonzy pushed a commit to horizonzy/pulsar that referenced this pull request Dec 17, 2023
lhotari pushed a commit that referenced this pull request Jan 26, 2024
lhotari pushed a commit that referenced this pull request Jan 26, 2024
Builds on: #20289

There are many certificates in our test code base. It would be much simpler to have one place were we create and manage certificates so that when we need to make changes, they are consolidated.

There is likely one or two more PRs to finish consolidating certs.

* Remove certs that are no longer used
* Replace references to old certs with references to the `certificate-authority` certs
* Create new server certs with valid hostnames on them so that tests will pass. Document the process used to create these certs.
* Fix an issue in the `PulsarTestContext` where the configuration was not correctly updated.
* Remove configurations that allow for insecure connections in tests that are doing some kind of TLS verification. The only places where we leave insecure validation in place is tests that are specifically verifying the functionality.
* Copy `certificate-authority` to the relevant `bouncy-castle` directory

When tests pass, this change will be correctly verified.

- [x] `doc`
This PR includes doc changes

PR in forked repository: michaeljmarshall#48

(cherry picked from commit d45a220)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/test cherry-picked/branch-3.0 doc-not-needed Your PR changes do not impact docs ready-to-test release/3.0.3 type/cleanup Code or doc cleanups e.g. remove the outdated documentation or remove the code no longer in use
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants