Refactor repository meta analyzer to use Protobuf schemas #411
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: nscuro <nscuro@protonmail.com>
Signed-off-by: nscuro <nscuro@protonmail.com>
... by creating new instances every time they're needed, through a factory. This is analog to how it works in vanilla DT (minus the factory). Previously, repository URLs were modified on `@ApplicationScoped` instances of the analyzers, which could cause race conditions and unexpected behavior. Signed-off-by: nscuro <nscuro@protonmail.com>
... and perform more extensive error handling. Signed-off-by: nscuro <nscuro@protonmail.com>
Signed-off-by: nscuro <nscuro@protonmail.com>
Signed-off-by: nscuro <nscuro@protonmail.com>
Signed-off-by: nscuro <nscuro@protonmail.com>
nscuro
force-pushed
the
repo-meta-analysis-proto
branch
from
6f94243
to
7265e67
Compare
![sonatype-lift[bot]](https://avatars.githubusercontent.com/in/9843?s=60&v=4){kind=small}
repository-meta-analyzer/src/main/java/org/hyades/processor/MetaAnalyzerProcessor.java
Outdated
Show resolved
Hide resolved
Signed-off-by: nscuro <nscuro@protonmail.com>
Signed-off-by: nscuro <nscuro@protonmail.com>
Merged
2 tasks
Signed-off-by: nscuro <nscuro@protonmail.com>
Signed-off-by: nscuro <nscuro@protonmail.com>
5 tasks
repository-meta-analyzer/src/main/java/org/hyades/processor/MetaAnalyzerProcessor.java
Show resolved
Hide resolved
Signed-off-by: nscuro <nscuro@protonmail.com>
nscuro
force-pushed
the
repo-meta-analysis-proto
branch
from
4dc0b01
to
3a0101b
Compare
|
Kudos, SonarCloud Quality Gate passed! |
VithikaS
approved these changes
Mar 17, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR refactors the repository meta analyzer to use Protobuf schemas.
While implementing this, I came across multiple issues with the current implementation, that I also addressed in this PR:
@ApplicationScopedCDI beans, meaning there would only ever be one instance per analyzer. Those instances were shared by all Kafka Streams stream threads. Analyzers are not thread safe however, as things like repository URLs and credentials are mutable fields of the analyzer classes.RepositoryMetaComponents on the API server side. That entity is identified purely by PURL type, namespace, and name. Keying the results by component UUID resulted in out-of-order consumption on the API server side, as results for the same type-namespace-name combination were spread across multiple partitions.Received repository meta information for X that is older than what's already in the database; DiscardinglogsOld Topology
As there is only one sub-topology, it is evident that no repartitioning was happening after the re-key operation. Splitting the stream based on the applicable analyzer did not have any effect (other than looking cool in the diagram).
New Topology
Drastically simplified by getting rid of the unnecessary split by analyzer type. We now have two sub-topologies due to the repartitioning.