Skip to content

Improvements to loading of compression configs#2538

Merged
ctubbsii merged 1 commit intoapache:mainfrom
ctubbsii:fix-compression-service-loading-and-hadoop-version
Mar 2, 2022
Merged

Improvements to loading of compression configs#2538
ctubbsii merged 1 commit intoapache:mainfrom
ctubbsii:fix-compression-service-loading-and-hadoop-version

Conversation

@ctubbsii
Copy link
Member

@ctubbsii ctubbsii commented Mar 2, 2022

This follows up from the work done in #2518 to remove the use of the
Java ServiceLoader for built-in known compression configs. Instead of
loading all compression configs through the ServiceLoader, this change
loads the known ones, and combines those with those discovered by the
ServiceLoader. This prevents issues during development where the
integration tests cannot be run inside an IDE, like Eclipse, due to the
auto-service annotation processor not having been executed in that
environment to make the implementing classes known to the ServiceLoader.
Now, they are always available for integration tests.

This also prevents the @AutoService annotation from being exposed in
SPI classes, although it's not clear that's necessarily a problem.

Additionally, this cleans up some of the previous code, that
unnecessarily placed the supported algorithm names in a list, then
converted it to an array, only to have it converted to a list again.

For the pom, this change also sets the auto-service version in a
property, so the multiple uses will use the same version, and explicitly
includes the annotation processor in the maven-compiler-plugin
configuration so it is consistent both within the errorprone profile and
outside it.

And finally, revert the version of Hadoop to 3.3.0, because 3.3.1 seems
to be the cause of some flaky testing. In particular, it seems to cause
unexpected EOF errors when trying to talk to data nodes, which is
revealed by regular CountNameNodeOpsBulkIT failures, some so bad they
seem to corrupt the XML output from failsafe, causing failsafe to be
unable to read the result of the test. I was not able to reproduce these
failures using 3.3.0, whereas I was seeing them regularly with 3.3.1.

This follows up from the work done in apache#2518 to remove the use of the
Java ServiceLoader for built-in known compression configs. Instead of
loading all compression configs through the ServiceLoader, this change
loads the known ones, and combines those with those discovered by the
ServiceLoader. This prevents issues during development where the
integration tests cannot be run inside an IDE, like Eclipse, due to the
auto-service annotation processor not having been executed in that
environment to make the implementing classes known to the ServiceLoader.
Now, they are always available for integration tests.

This also prevents the `@AutoService` annotation from being exposed in
SPI classes, although it's not clear that's necessarily a problem.

Additionally, this cleans up some of the previous code, that
unnecessarily placed the supported algorithm names in a list, then
converted it to an array, only to have it converted to a list again.

For the pom, this change also sets the auto-service version in a
property, so the multiple uses will use the same version, and explicitly
includes the annotation processor in the maven-compiler-plugin
configuration so it is consistent both within the errorprone profile and
outside it.

And finally, revert the version of Hadoop to 3.3.0, because 3.3.1 seems
to be the cause of some flaky testing. In particular, it seems to cause
unexpected EOF errors when trying to talk to data nodes, which is
revealed by regular CountNameNodeOpsBulkIT failures, some so bad they
seem to corrupt the XML output from failsafe, causing failsafe to be
unable to read the result of the test. I was not able to reproduce these
failures using 3.3.0, whereas I was seeing them regularly with 3.3.1.
@ctubbsii ctubbsii requested a review from dlmarion March 2, 2022 06:34
@ctubbsii ctubbsii self-assigned this Mar 2, 2022
Copy link
Contributor

@dlmarion dlmarion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ctubbsii ctubbsii merged commit 48679fe into apache:main Mar 2, 2022
@ctubbsii ctubbsii deleted the fix-compression-service-loading-and-hadoop-version branch March 2, 2022 15:19
@ctubbsii ctubbsii added this to the 2.1.0 milestone Jul 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants