Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Scala Common Enrich: separate EnrichmentRegistry parsing from its construction #4033

Closed
BenFradet opened this issue Apr 10, 2019 · 3 comments
Assignees

Comments

@BenFradet
Copy link
Contributor

BenFradet commented Apr 10, 2019

the workflow for an enrichment platform should be:

  1. parse and validate enrichment jsons: Json -> F[List[EnrichmentConf]] (enrichment configs contain the files to cache)
  2. cache files: List[EnrichmentConf] -> F[Unit] or Unit, this is up to the enricher
  3. construct EnrichmentRegistry: List[EnrichmentConf] -> F[List[Enrichment]]
  4. enrich: List[Enrichment] -> F[List[EnrichedEvent]]

We need to separate parsing from construction because some enrichments' constructions are side-effectful, for example referer parser which reads a file

@BenFradet BenFradet self-assigned this Apr 10, 2019
@szareiangm
Copy link
Contributor

can we have step3 produce [EnrichmentConf] -> F[List[Enrichment]]? It would help to have multiple API Request Enrichments

@BenFradet
Copy link
Contributor Author

I added List instead of [] so it's clearer, however supporting multiple api request enrichments is not planned for this release.

@szareiangm
Copy link
Contributor

That's okay. At least I will use the result of this issue for my issue. It would be ways more straight-forward.

benjben pushed a commit that referenced this issue Aug 6, 2019
chuwy pushed a commit that referenced this issue Nov 5, 2019
chuwy pushed a commit that referenced this issue Nov 5, 2019
chuwy pushed a commit that referenced this issue Nov 5, 2019
chuwy pushed a commit that referenced this issue Nov 6, 2019
chuwy pushed a commit that referenced this issue Nov 6, 2019
chuwy pushed a commit that referenced this issue Nov 22, 2019
benjben pushed a commit that referenced this issue Dec 6, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants