From 926004c248685ca09013e516ed1b7a4c2a6ec3b2 Mon Sep 17 00:00:00 2001 From: Sven Strittmatter Date: Tue, 23 Jun 2020 18:14:54 +0200 Subject: [PATCH 01/17] #30 Use literal and source blocks for diagram Asciidoc uses a different syntax for code or monospaced blocks. Also remove leading white spaces. Signed-off-by: Sven Strittmatter --- docs/adr/adr_0001.adoc | 49 +++++++++++++++++++++++------------------- 1 file changed, 27 insertions(+), 22 deletions(-) diff --git a/docs/adr/adr_0001.adoc b/docs/adr/adr_0001.adoc index c6c898bf..00d26f9b 100644 --- a/docs/adr/adr_0001.adoc +++ b/docs/adr/adr_0001.adoc @@ -16,7 +16,8 @@ | Author(s) | Jannik Hollenbach , Jorge Estigarribia , - Robert Seedorff + Robert Seedorff , + Sven Strittmatter |==== == Context @@ -24,17 +25,19 @@ === Status Quo One major challenge implementing the _secureCodeBox_ is to provide a flexible and modular architecture, which enables the open source community to easily understand the concepts and especially to extend the _secureCodeBox_ with individual features. Therefore we decided to seperate the process stages of a single security scan (instance of scanType CRD) in 3 major phases: -``` - ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ - │ Scanning ├─────────▶│ Parsing ├─────────▶│ Persisting │ - │ (Phase 1) │ │ (Phase 2) │ │ (Phase 3) │ - └──────────────────┘ └──────────────────┘ └──────────────────┘ -``` + +.... +┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ +│ Scanning ├─────────▶│ Parsing ├─────────▶│ Persisting │ +│ (Phase 1) │ │ (Phase 2) │ │ (Phase 3) │ +└──────────────────┘ └──────────────────┘ └──────────────────┘ +.... + By now the "Persisting Phase 3" was implemented by so called _persistenceProviders_ e.g the *persistence-elastic* provider which is responsible for persisting all findings in a given elasticsearch database. The _secureCodeBox_ Operator is aware of this 3 phases and is responsible for the state model and execution of each security scan (instance of scanType CRD). === Problem and Question -We identified different additional UseCases with a more _data proccessing oriented_ pattern than the implemented _persisting phase3_ indicates. For example, we implemented a so called "MetaDataProvider" feature, which is responsible for enhancing each security finding with additional metadata. But the MetaDataProvider must be executed after the _parsing phase 2_ and before the _persisting phase 3_ because it depends on the parsed finding results (which will be enhanced) and the update findings should be also persisted. +We identified different additional UseCases with a more _data proccessing oriented_ pattern than the implemented _persisting phase3_ indicates. For example, we implemented a so called "MetaDataProvider" feature, which is responsible for enhancing each security finding with additional metadata. But the MetaDataProvider must be executed after the _parsing phase 2_ and before the _persisting phase 3_ because it depends on the parsed finding results (which will be enhanced) and the update findings should be also persisted. To find a proper solution, we splitted the topic into the follwong two questions: @@ -43,6 +46,7 @@ To find a proper solution, we splitted the topic into the follwong two questions * Question 2: How should the execution model look like for each concept? ==== Question 1: Should we unify the concepts MetaDataProvider and PersistenceProvider? + ==== Solution approach 1: Unify Both "modules" are "processing" the security findings, which are generated in the parsing phase. @@ -68,7 +72,8 @@ These could be implemented with a `type` attribute, which declares if they are ` The operator would process all these resources in the namespace and execute the `ReadAndWrite` ones first (in serial: one at a time to avoid write conflicts) and then the `ReadOnly` ones (in parallel). -```yaml +[source,yaml] +---- apiVersion: execution.experimental.securecodebox.io/v1 kind: ScanCompletionHook metadata: @@ -77,11 +82,11 @@ spec: type: ReadAndWrite # If implemented like the current persistence provider image: my-metadata:v2.0.0 -``` +---- The Execution Flow would then look something like this: -``` +.... ┌ ReadOnly─Hooks─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┌ ReadAndWriteHooks ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┌────────────────────────────────┐ │ ┌───────────────────────┐ │ ┌──┼▶│ Elastic PersistenceProvider │ @@ -90,7 +95,7 @@ The Execution Flow would then look something like this: └──────────────────┘ └──────────────────┘ │ └───────────────────────┘ └───────────────────────┘ └───▶│ DefectDojo PersistenceProvider │ │ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ │ └────────────────────────────────┘ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ -``` +.... **Pros:** @@ -107,7 +112,7 @@ The Execution Flow would then look something like this: Keep PersistenceProvider as they are and introduce new "MetaDataProvider" CRD which gets executed before the PersistenceProviders by the operator. -``` +.... ┌ Persistence Provider─ ─ ─ ─ ─ ─ ─ ─ ┌ MetaData Provider ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┌────────────────────────────────┐ │ ┌───────────────────────┐ │ ┌──┼▶│ Elastic PersistenceProvider │ @@ -116,7 +121,7 @@ Keep PersistenceProvider as they are and introduce new "MetaDataProvider" CRD wh └──────────────────┘ └──────────────────┘ │ └───────────────────────┘ └───────────────────────┘ └───▶│ DefectDojo PersistenceProvider │ │ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ │ └────────────────────────────────┘ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ -``` +.... **Pros:** @@ -180,17 +185,17 @@ Https server receiving findings and returning results. Regarding the Question 1 it seems that both solution approaches are resulting in the same execution model. We descided to implement solution approach 1 and unify both concepts into a more general concept with the name _"hook concept"_. Therefore we exchange the existing name `persistenceProvider` for phase 3 in the excecution model with a more general term `DataProcessing`: -``` - ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ - │ Scanning ├─────────▶│ Parsing ├─────────▶│ DataProcessing │ - │ (Phase 1) │ │ (Phase 2) │ │ (Phase 3) │ - └──────────────────┘ └──────────────────┘ └──────────────────┘ -``` +.... +┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ +│ Scanning ├─────────▶│ Parsing ├─────────▶│ DataProcessing │ +│ (Phase 1) │ │ (Phase 2) │ │ (Phase 3) │ +└──────────────────┘ └──────────────────┘ └──────────────────┘ +.... Regarding the Question 2 we decided to implement the solution approach 1 with a job-based approach (no active service componend needed). The Phase 3 `DataProcessing` will be therefore splitt into to seperate phases named `ReadAndWriteHooks (3.1)` and `ReadOnlyHooks (3.2)` -``` +.... ┌ DataProcessing: ReadOnlyHooks ─ ─ ─ ┌ DataProcessing: ReadAndWriteHooks ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┌────────────────────────────────┐ │ ┌───────────────────────┐ │ ┌──┼▶│ Elastic PersistenceProvider │ @@ -199,7 +204,7 @@ The Phase 3 `DataProcessing` will be therefore splitt into to seperate phases na └──────────────────┘ └──────────────────┘ │ └───────────────────────┘ └───────────────────────┘ └───▶│ DefectDojo PersistenceProvider │ │ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ │ └────────────────────────────────┘ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ -``` +.... == Consequences From 2005d079203a95154d263e5aadbaf6c473e1cda4 Mon Sep 17 00:00:00 2001 From: Sven Strittmatter Date: Tue, 23 Jun 2020 18:17:46 +0200 Subject: [PATCH 02/17] #30 Only use one style for lists Signed-off-by: Sven Strittmatter --- docs/adr/adr_0001.adoc | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/docs/adr/adr_0001.adoc b/docs/adr/adr_0001.adoc index 00d26f9b..b8d93e6e 100644 --- a/docs/adr/adr_0001.adoc +++ b/docs/adr/adr_0001.adoc @@ -64,9 +64,9 @@ The are two variants: We could do a similar thing and introduce CRD which allows to execute "custom code" (generally speaking, depends on the second question) after a scan has completed (meaning both scan and parser phases are done). Some name ideas: -- `ScanHooks` -- `ScanCompletionHooks` -- `FindingProcessors` +* `ScanHooks` +* `ScanCompletionHooks` +* `FindingProcessors` These could be implemented with a `type` attribute, which declares if they are `ReadOnly` or `ReadAndWrite`. @@ -99,14 +99,14 @@ The Execution Flow would then look something like this: **Pros:** -- Only one Implementation -- Pretty Generic to expand and test out new ideas without having to modify the secureCodeBox Operator +* Only one Implementation +* Pretty Generic to expand and test out new ideas without having to modify the secureCodeBox Operator **Cons:** -- Possible "over-abstraction"? -- Need to refactor the ElasticSearch PersistenceProvider -- The "General Implementation" will be harder than the individual ones +* Possible "over-abstraction"? +* Need to refactor the ElasticSearch PersistenceProvider +* The "General Implementation" will be harder than the individual ones ==== Solution approach 1: Keep Split between Persistence Provider and MetaData Provider @@ -125,12 +125,12 @@ Keep PersistenceProvider as they are and introduce new "MetaDataProvider" CRD wh **Pros:** -- Quicker to implement -- Might be worth it to have a seperate concept for it +* Quicker to implement +* Might be worth it to have a seperate concept for it **Cons:** -- Really worth introducing a new CRD for everything, especially when the are conceptually pretty close? +* Really worth introducing a new CRD for everything, especially when the are conceptually pretty close? === Question 2: How should the execution model look like for each? @@ -211,6 +211,7 @@ The Phase 3 `DataProcessing` will be therefore splitt into to seperate phases na With the new `Hook Concept` we open the `DataProcessing` Phase 3 to a more intuitive and flexible architecture. It is easier to understand because _WebHooks_ are already a well known concept. It is possible to keep the existing implementation of the `persistenceProviders` and to integrate them with a lot of other possible data processing components in a more general fashion. In the end, this step will result in a lot of additional feature possibilities, which go fare beyond the existing ones. Therefore we only need to implement this concept once in the secureCodeBox Operator and new ideas for extending the DataProcessing will not enforce conceptual or architectural changes. Ideas for additional data processing hooks: + * Notifier-Hooks (ReadOnlyHook) e.g. for chat systems (slack, teams...) or metric / alerting systems * MetaData Enrichment Hooks (ReadAndWriteHook) * FilterData Hooks (e.g. false/positive Handling) (ReadAndWriteHook) From c3dd85333c99a1b04e16d7a2d3f79e66a4a8736a Mon Sep 17 00:00:00 2001 From: Sven Strittmatter Date: Tue, 23 Jun 2020 18:18:40 +0200 Subject: [PATCH 03/17] #30 Numerate the enumerated variants Signed-off-by: Sven Strittmatter --- docs/adr/adr_0001.adoc | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/adr/adr_0001.adoc b/docs/adr/adr_0001.adoc index b8d93e6e..49926a9b 100644 --- a/docs/adr/adr_0001.adoc +++ b/docs/adr/adr_0001.adoc @@ -42,8 +42,8 @@ We identified different additional UseCases with a more _data proccessing orient To find a proper solution, we splitted the topic into the follwong two questions: -* Question 1: Should we unify the concepts MetaDataProvider and PersistenceProvider? -* Question 2: How should the execution model look like for each concept? +. Question 1: Should we unify the concepts MetaDataProvider and PersistenceProvider? +. Question 2: How should the execution model look like for each concept? ==== Question 1: Should we unify the concepts MetaDataProvider and PersistenceProvider? @@ -59,8 +59,8 @@ There is a similar thing in kubernetes called [AdmissionController](https://kube The are two variants: -* `ValidatingWebhookConfiguration` (ReadOnly) *Executed Last* -* `MutatingWebhookConfiguration` (ReadAndWrite) *Executed First* +. `ValidatingWebhookConfiguration` (ReadOnly) *Executed Last* +. `MutatingWebhookConfiguration` (ReadAndWrite) *Executed First* We could do a similar thing and introduce CRD which allows to execute "custom code" (generally speaking, depends on the second question) after a scan has completed (meaning both scan and parser phases are done). Some name ideas: From eb0d1d0aeb285f6a658e895a261d109f78af1c01 Mon Sep 17 00:00:00 2001 From: Sven Strittmatter Date: Tue, 23 Jun 2020 18:20:30 +0200 Subject: [PATCH 04/17] #30 Remove redundnat text By the ordered lists numbers it became obvious that they are questioon one and two. Signed-off-by: Sven Strittmatter --- docs/adr/adr_0001.adoc | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/adr/adr_0001.adoc b/docs/adr/adr_0001.adoc index 49926a9b..315fd122 100644 --- a/docs/adr/adr_0001.adoc +++ b/docs/adr/adr_0001.adoc @@ -39,11 +39,10 @@ By now the "Persisting Phase 3" was implemented by so called _persistenceProvide We identified different additional UseCases with a more _data proccessing oriented_ pattern than the implemented _persisting phase3_ indicates. For example, we implemented a so called "MetaDataProvider" feature, which is responsible for enhancing each security finding with additional metadata. But the MetaDataProvider must be executed after the _parsing phase 2_ and before the _persisting phase 3_ because it depends on the parsed finding results (which will be enhanced) and the update findings should be also persisted. - To find a proper solution, we splitted the topic into the follwong two questions: -. Question 1: Should we unify the concepts MetaDataProvider and PersistenceProvider? -. Question 2: How should the execution model look like for each concept? +. Should we unify the concepts MetaDataProvider and PersistenceProvider? +. How should the execution model look like for each concept? ==== Question 1: Should we unify the concepts MetaDataProvider and PersistenceProvider? From c87bff9d3212c1ef8bbc2ce81913adc46f149b40 Mon Sep 17 00:00:00 2001 From: Sven Strittmatter Date: Tue, 23 Jun 2020 18:27:06 +0200 Subject: [PATCH 05/17] #30 Fixing the headline levels The levels were slightly mangled up. Also it looks nicer to denote the pro/cons also as headlines. One question lasts: Where is option 1? Because there is a option 2 headline. Signed-off-by: Sven Strittmatter --- docs/adr/adr_0001.adoc | 29 ++++++++++++++--------------- 1 file changed, 14 insertions(+), 15 deletions(-) diff --git a/docs/adr/adr_0001.adoc b/docs/adr/adr_0001.adoc index 315fd122..0567cb14 100644 --- a/docs/adr/adr_0001.adoc +++ b/docs/adr/adr_0001.adoc @@ -46,7 +46,7 @@ To find a proper solution, we splitted the topic into the follwong two questions ==== Question 1: Should we unify the concepts MetaDataProvider and PersistenceProvider? -==== Solution approach 1: Unify +===== Solution approach 1: Unify Both "modules" are "processing" the security findings, which are generated in the parsing phase. But there is one larger difference between them: @@ -96,18 +96,18 @@ The Execution Flow would then look something like this: ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ .... -**Pros:** +====== Pros * Only one Implementation * Pretty Generic to expand and test out new ideas without having to modify the secureCodeBox Operator -**Cons:** +====== Cons * Possible "over-abstraction"? * Need to refactor the ElasticSearch PersistenceProvider * The "General Implementation" will be harder than the individual ones -==== Solution approach 1: Keep Split between Persistence Provider and MetaData Provider +===== Solution approach 1: Keep Split between Persistence Provider and MetaData Provider Keep PersistenceProvider as they are and introduce new "MetaDataProvider" CRD which gets executed before the PersistenceProviders by the operator. @@ -122,18 +122,18 @@ Keep PersistenceProvider as they are and introduce new "MetaDataProvider" CRD wh ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ .... -**Pros:** +====== Pros * Quicker to implement * Might be worth it to have a seperate concept for it -**Cons:** +====== Cons * Really worth introducing a new CRD for everything, especially when the are conceptually pretty close? -=== Question 2: How should the execution model look like for each? +==== Question 2: How should the execution model look like for each? -==== Solution approach 1: Like the persistence provider +===== Solution approach 1: Like the persistence provider Basically a docker container which process takes two command line args: @@ -146,31 +146,30 @@ Examples: * java `java my-metadata.jar "https://storage.googleapi.com/..." "https://storage.googleapi.com/..."` * golang `./my-metadata "https://storage.googleapi.com/..." "https://storage.googleapi.com/..."` -**Pros:** +====== Pros * on liner with the current implementations * code overhead / wrapper code is pretty minimal * zero scale - no resource costs when nothing is running -**Cons:** +===== Cons * results in too many k8s jobs? ** resource blocking on finished resources ** ttlAfterFinished enabled * container runtime overhead (especially time) -### Option 2: A WebHooks like concept +=== Option 2: A WebHooks like concept -Analog to kubernetes webhooks. -Https server receiving findings and returning results. +Analog to kubernetes webhooks. Https server receiving findings and returning results. -**Pros:** +==== Pros * MilliSeconds instead of seconds for processing * No ContainerCreation Overhead * No additional k8s jobs needed -**Cons:** +===== Cons * Introduces new running Services that need to be maintained and have uptime * Code Overhead / Boilerplate (Can be mitigated by sdk) From 6ff9c3dc74204e8ce26649fbc48c3589331e8b0e Mon Sep 17 00:00:00 2001 From: Sven Strittmatter Date: Tue, 23 Jun 2020 18:30:27 +0200 Subject: [PATCH 06/17] #30 Fixes the headline cases Signed-off-by: Sven Strittmatter --- docs/adr/adr_0001.adoc | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/adr/adr_0001.adoc b/docs/adr/adr_0001.adoc index 0567cb14..4aa7c30a 100644 --- a/docs/adr/adr_0001.adoc +++ b/docs/adr/adr_0001.adoc @@ -44,9 +44,9 @@ To find a proper solution, we splitted the topic into the follwong two questions . Should we unify the concepts MetaDataProvider and PersistenceProvider? . How should the execution model look like for each concept? -==== Question 1: Should we unify the concepts MetaDataProvider and PersistenceProvider? +==== Question 1: Should We Unify the Concepts MetaDataProvider and PersistenceProvider? -===== Solution approach 1: Unify +===== Solution Approach 1: Unify Both "modules" are "processing" the security findings, which are generated in the parsing phase. But there is one larger difference between them: @@ -107,7 +107,7 @@ The Execution Flow would then look something like this: * Need to refactor the ElasticSearch PersistenceProvider * The "General Implementation" will be harder than the individual ones -===== Solution approach 1: Keep Split between Persistence Provider and MetaData Provider +===== Solution Approach 1: Keep Split between Persistence Provider and MetaData Provider Keep PersistenceProvider as they are and introduce new "MetaDataProvider" CRD which gets executed before the PersistenceProviders by the operator. @@ -131,9 +131,9 @@ Keep PersistenceProvider as they are and introduce new "MetaDataProvider" CRD wh * Really worth introducing a new CRD for everything, especially when the are conceptually pretty close? -==== Question 2: How should the execution model look like for each? +==== Question 2: How Should the Execution Model Look like for Each? -===== Solution approach 1: Like the persistence provider +===== Solution Approach 1: Like the Persistence Provider Basically a docker container which process takes two command line args: @@ -159,7 +159,7 @@ Examples: ** ttlAfterFinished enabled * container runtime overhead (especially time) -=== Option 2: A WebHooks like concept +=== Option 2: A WebHooks Like Concept Analog to kubernetes webhooks. Https server receiving findings and returning results. From de48e642d9115e5c83b9930ea5a36aa9e87e3156 Mon Sep 17 00:00:00 2001 From: Sven Strittmatter Date: Tue, 23 Jun 2020 20:10:31 +0200 Subject: [PATCH 07/17] #30 Run a spellchecker on ADR 1 Signed-off-by: Sven Strittmatter --- docs/adr/adr_0001.adoc | 43 ++++++++++++++++++++---------------------- 1 file changed, 20 insertions(+), 23 deletions(-) diff --git a/docs/adr/adr_0001.adoc b/docs/adr/adr_0001.adoc index 4aa7c30a..0b68cc03 100644 --- a/docs/adr/adr_0001.adoc +++ b/docs/adr/adr_0001.adoc @@ -1,11 +1,8 @@ [[ADR-0000]] -= ADR-0000: How can we introduce a more general extension concept for data proccessing modules? += ADR-0000: How can we introduce a more general extension concept for data processing modules? [cols="h,d",grid=rows,frame=none,stripes=none,caption="Status",%autowidth] |==== -// Use one of the ADR status parameter based on status -// Please add a cross reference link to the new ADR on 'superseded' ADR. -// e.g.: {adr_suposed_by} <> | Status | ACCEPTED @@ -24,7 +21,7 @@ === Status Quo -One major challenge implementing the _secureCodeBox_ is to provide a flexible and modular architecture, which enables the open source community to easily understand the concepts and especially to extend the _secureCodeBox_ with individual features. Therefore we decided to seperate the process stages of a single security scan (instance of scanType CRD) in 3 major phases: +One major challenge implementing the _secureCodeBox_ is to provide a flexible and modular architecture, which enables the open source community to easily understand the concepts and especially to extend the _secureCodeBox_ with individual features. Therefore we decided to separate the process stages of a single security scan (instance of scanType CRD) in 3 major phases: .... ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ @@ -37,9 +34,9 @@ By now the "Persisting Phase 3" was implemented by so called _persistenceProvide === Problem and Question -We identified different additional UseCases with a more _data proccessing oriented_ pattern than the implemented _persisting phase3_ indicates. For example, we implemented a so called "MetaDataProvider" feature, which is responsible for enhancing each security finding with additional metadata. But the MetaDataProvider must be executed after the _parsing phase 2_ and before the _persisting phase 3_ because it depends on the parsed finding results (which will be enhanced) and the update findings should be also persisted. +We identified different additional UseCases with a more _data processing oriented_ pattern than the implemented _persisting phase3_ indicates. For example, we implemented a so called "MetaDataProvider" feature, which is responsible for enhancing each security finding with additional metadata. But the MetaDataProvider must be executed after the _parsing phase 2_ and before the _persisting phase 3_ because it depends on the parsed finding results (which will be enhanced) and the update findings should be also persisted. -To find a proper solution, we splitted the topic into the follwong two questions: +To find a proper solution, we split the topic into the following two questions: . Should we unify the concepts MetaDataProvider and PersistenceProvider? . How should the execution model look like for each concept? @@ -51,8 +48,8 @@ To find a proper solution, we splitted the topic into the follwong two questions Both "modules" are "processing" the security findings, which are generated in the parsing phase. But there is one larger difference between them: -* `PersistenceProvider` is proccesing the findings with a *ReadOnly* pattern -* `MetaDataProvider` is proccesing the findings with a *ReadAndWrite* pattern +* `PersistenceProvider` is processing the findings with a *ReadOnly* pattern +* `MetaDataProvider` is processing the findings with a *ReadAndWrite* pattern There is a similar thing in kubernetes called [AdmissionController](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/), with the exception that the are executed before a resource gets created. @@ -125,7 +122,7 @@ Keep PersistenceProvider as they are and introduce new "MetaDataProvider" CRD wh ====== Pros * Quicker to implement -* Might be worth it to have a seperate concept for it +* Might be worth it to have a separate concept for it ====== Cons @@ -142,9 +139,9 @@ Basically a docker container which process takes two command line args: Examples: -* Node.js `node my-metadata.js "https://storage.googleapi.com/..." "https://storage.googleapi.com/..."` -* java `java my-metadata.jar "https://storage.googleapi.com/..." "https://storage.googleapi.com/..."` -* golang `./my-metadata "https://storage.googleapi.com/..." "https://storage.googleapi.com/..."` +* NodeJS: `node my-metadata.js "https://storage.googleapi.com/..." "https://storage.googleapi.com/..."` +* Java: `java my-metadata.jar "https://storage.googleapi.com/..." "https://storage.googleapi.com/..."` +* Golang: `./my-metadata "https://storage.googleapi.com/..." "https://storage.googleapi.com/..."` ====== Pros @@ -172,16 +169,16 @@ Analog to kubernetes webhooks. Https server receiving findings and returning res ===== Cons * Introduces new running Services that need to be maintained and have uptime -* Code Overhead / Boilerplate (Can be mitigated by sdk) +* Code Overhead / Boilerplate (Can be mitigated by SDK) * Debugging of individual MetaDataProvider is harder as everything is handled by a single service * Introduces "New" Concept * Certificate Management for webhook services (`cert-manager` required by default?) * Scaling for systems with lots of load could be a problem -* One service per namespace (multiple tenants) needed => results in many running active services which is ressource consuming +* One service per namespace (multiple tenants) needed => results in many running active services which is resource consuming == Decision -Regarding the Question 1 it seems that both solution approaches are resulting in the same execution model. We descided to implement solution approach 1 and unify both concepts into a more general concept with the name _"hook concept"_. Therefore we exchange the existing name `persistenceProvider` for phase 3 in the excecution model with a more general term `DataProcessing`: +Regarding the Question 1 it seems that both solution approaches are resulting in the same execution model. We decided to implement solution approach 1 and unify both concepts into a more general concept with the name _"hook concept"_. Therefore we exchange the existing name `persistenceProvider` for phase 3 in the execution model with a more general term `DataProcessing`: .... ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ @@ -190,8 +187,8 @@ Regarding the Question 1 it seems that both solution approaches are resulting in └──────────────────┘ └──────────────────┘ └──────────────────┘ .... -Regarding the Question 2 we decided to implement the solution approach 1 with a job-based approach (no active service componend needed). -The Phase 3 `DataProcessing` will be therefore splitt into to seperate phases named `ReadAndWriteHooks (3.1)` and `ReadOnlyHooks (3.2)` +Regarding the Question 2 we decided to implement the solution approach 1 with a job-based approach (no active service component needed). +The Phase 3 `DataProcessing` will be therefore split into to separate phases named `ReadAndWriteHooks (3.1)` and `ReadOnlyHooks (3.2)` .... ┌ DataProcessing: ReadOnlyHooks ─ ─ ─ @@ -210,8 +207,8 @@ With the new `Hook Concept` we open the `DataProcessing` Phase 3 to a more intui Ideas for additional data processing hooks: -* Notifier-Hooks (ReadOnlyHook) e.g. for chat systems (slack, teams...) or metric / alerting systems -* MetaData Enrichment Hooks (ReadAndWriteHook) -* FilterData Hooks (e.g. false/positive Handling) (ReadAndWriteHook) -* SystemIntegration Hooks (ReadOnlyHook) e.g. for ticketing systems like JIRA -* CascadingScans Hooks (ReadOnlyHook) e.g. for starting new security scans based on findings +* Notifier-Hooks (ReadOnlyHook) e.g., for chat (slack, teams etc.), metric, alerting systems +* MetaData enrichment hooks (ReadAndWriteHook) +* FilterData Hooks (e.g., false/positive handling) (ReadAndWriteHook) +* SystemIntegration Hooks (ReadOnlyHook) e.g., for ticketing systems like Jira +* CascadingScans Hooks (ReadOnlyHook) e.g., for starting new security scans based on findings From aea59d7a97e19519d72e17e52dd0d0126600666a Mon Sep 17 00:00:00 2001 From: Sven Strittmatter Date: Mon, 29 Jun 2020 11:41:25 +0200 Subject: [PATCH 08/17] #30 Slightly refined the status quo paragraph Signed-off-by: Sven Strittmatter --- docs/adr/adr_0001.adoc | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/adr/adr_0001.adoc b/docs/adr/adr_0001.adoc index 0b68cc03..5c68fc5b 100644 --- a/docs/adr/adr_0001.adoc +++ b/docs/adr/adr_0001.adoc @@ -21,16 +21,16 @@ === Status Quo -One major challenge implementing the _secureCodeBox_ is to provide a flexible and modular architecture, which enables the open source community to easily understand the concepts and especially to extend the _secureCodeBox_ with individual features. Therefore we decided to separate the process stages of a single security scan (instance of scanType CRD) in 3 major phases: +One major challenge implementing the _secureCodeBox_ is to provide a flexible and modular architecture, which enables the open source community to easily understand the concepts and especially to extend the _secureCodeBox_ with individual features. Therefore we decided to separate the process stages of a single security scan (instance of _scanType_ custom resource definition; further abbreviated with _CRD_) in three major phases: .... ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ -│ Scanning ├─────────▶│ Parsing ├─────────▶│ Persisting │ -│ (Phase 1) │ │ (Phase 2) │ │ (Phase 3) │ +│ scanning ├─────────▶│ parsing ├─────────▶│ persisting │ +│ (phase 1) │ │ (phase 2) │ │ (phase 3) │ └──────────────────┘ └──────────────────┘ └──────────────────┘ .... -By now the "Persisting Phase 3" was implemented by so called _persistenceProviders_ e.g the *persistence-elastic* provider which is responsible for persisting all findings in a given elasticsearch database. The _secureCodeBox_ Operator is aware of this 3 phases and is responsible for the state model and execution of each security scan (instance of scanType CRD). +By now the phase 3 "`persisting`" was implemented by so called _persistenceProviders_ (e.g. the _persistence-elastic_ provider which is responsible for persisting all findings in a given elasticsearch database). The _secureCodeBox_ Operator is aware of this 3 phases and is responsible for the state model and execution of each security scan. === Problem and Question From e2c6a17ab506c77827c11f50c319c04c22526a40 Mon Sep 17 00:00:00 2001 From: Sven Strittmatter Date: Mon, 29 Jun 2020 11:45:55 +0200 Subject: [PATCH 09/17] #30 Use a consequent style of quotes and emphasise - Quoting only the names of thethree phases from the graph. May we shouldemphasize them too. - Emphasize everything which is a special named thing in SCB. - Also fixes casing: Only names start with capital letter. Signed-off-by: Sven Strittmatter --- docs/adr/adr_0001.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/adr/adr_0001.adoc b/docs/adr/adr_0001.adoc index 5c68fc5b..63d48468 100644 --- a/docs/adr/adr_0001.adoc +++ b/docs/adr/adr_0001.adoc @@ -30,11 +30,11 @@ One major challenge implementing the _secureCodeBox_ is to provide a flexible an └──────────────────┘ └──────────────────┘ └──────────────────┘ .... -By now the phase 3 "`persisting`" was implemented by so called _persistenceProviders_ (e.g. the _persistence-elastic_ provider which is responsible for persisting all findings in a given elasticsearch database). The _secureCodeBox_ Operator is aware of this 3 phases and is responsible for the state model and execution of each security scan. +By now the phase 3 "`persisting`" was implemented by so called _persistenceProviders_ (e.g., the _persistence-elastic_ provider which is responsible for persisting all findings in a given elasticsearch database). The _secureCodeBox_ Operator is aware of this 3 phases and is responsible for the state model and execution of each security scan. === Problem and Question -We identified different additional UseCases with a more _data processing oriented_ pattern than the implemented _persisting phase3_ indicates. For example, we implemented a so called "MetaDataProvider" feature, which is responsible for enhancing each security finding with additional metadata. But the MetaDataProvider must be executed after the _parsing phase 2_ and before the _persisting phase 3_ because it depends on the parsed finding results (which will be enhanced) and the update findings should be also persisted. +We identified different additional use cases with a more "`data processing oriented`" pattern than the implemented phase 3 "`persisting`" indicates. For example, we implemented a so called _MetaDataProvider_ feature, which is responsible for enhancing each security finding with additional metadata. But the _MetaDataProvider_ must be executed after the phase 2 "`parsing`" and before the phase 3 "`persisting`" because it depends on the parsed finding results (which will be enhanced) and the updated findings should be also persisted. To find a proper solution, we split the topic into the following two questions: From b9ee818a04710d45a2a079d3284b08b23abfe597 Mon Sep 17 00:00:00 2001 From: Sven Strittmatter Date: Mon, 29 Jun 2020 12:26:19 +0200 Subject: [PATCH 10/17] #30 Adapt the quesion 1 paragraph to the styling of the previous one Signed-off-by: Sven Strittmatter --- docs/adr/adr_0001.adoc | 45 ++++++++++++++++++++++-------------------- 1 file changed, 24 insertions(+), 21 deletions(-) diff --git a/docs/adr/adr_0001.adoc b/docs/adr/adr_0001.adoc index 63d48468..900fc6ec 100644 --- a/docs/adr/adr_0001.adoc +++ b/docs/adr/adr_0001.adoc @@ -30,7 +30,7 @@ One major challenge implementing the _secureCodeBox_ is to provide a flexible an └──────────────────┘ └──────────────────┘ └──────────────────┘ .... -By now the phase 3 "`persisting`" was implemented by so called _persistenceProviders_ (e.g., the _persistence-elastic_ provider which is responsible for persisting all findings in a given elasticsearch database). The _secureCodeBox_ Operator is aware of this 3 phases and is responsible for the state model and execution of each security scan. +By now the phase 3 "`persisting`" was implemented by so called _PersistenceProviders_ (e.g., the _persistence-elastic_ provider which is responsible for persisting all findings in a given elasticsearch database). The _secureCodeBox_ Operator is aware of this 3 phases and is responsible for the state model and execution of each security scan. === Problem and Question @@ -38,35 +38,38 @@ We identified different additional use cases with a more "`data processing orien To find a proper solution, we split the topic into the following two questions: -. Should we unify the concepts MetaDataProvider and PersistenceProvider? +. Should we unify the concepts _MetaDataProvider_ and _PersistenceProvider_? . How should the execution model look like for each concept? ==== Question 1: Should We Unify the Concepts MetaDataProvider and PersistenceProvider? ===== Solution Approach 1: Unify -Both "modules" are "processing" the security findings, which are generated in the parsing phase. -But there is one larger difference between them: +Both "`modules`" are "`processing`" the security findings, which were generated in the phase 2 "`parsing`", +but there is one major difference between them: -* `PersistenceProvider` is processing the findings with a *ReadOnly* pattern -* `MetaDataProvider` is processing the findings with a *ReadAndWrite* pattern +* a _PersistenceProvider_ is processing the findings *read only*, and +* a _MetaDataProvider_ is processing the findings *read and write*. -There is a similar thing in kubernetes called [AdmissionController](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/), with the exception that the are executed before a resource gets created. +There is a similar concept in Kubernetes called https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/[AdmissionController], but with the exception that the will be executed before a resource is created. -The are two variants: +There are two variants of _AdmissionControllers_: -. `ValidatingWebhookConfiguration` (ReadOnly) *Executed Last* -. `MutatingWebhookConfiguration` (ReadAndWrite) *Executed First* +. _ValidatingWebhookConfiguration_: *read only*, *executed last*; and +. _MutatingWebhookConfiguration_: *read and write*, *executed first*. -We could do a similar thing and introduce CRD which allows to execute "custom code" (generally speaking, depends on the second question) after a scan has completed (meaning both scan and parser phases are done). Some name ideas: +We could do a similar thing and introduce CRD which allows to execute "`custom code`" (depends on the second question) after a scan has completed (meaning both phases "`scan`" and "`parsing`" were done). Some name ideas: -* `ScanHooks` -* `ScanCompletionHooks` -* `FindingProcessors` +* _ScanHooks_ +* _ScanCompletionHooks_ +* _FindingProcessors_ -These could be implemented with a `type` attribute, which declares if they are `ReadOnly` or `ReadAndWrite`. +These could be implemented with a `type` attribute, which declares if they are *read only* or *read and write*. -The operator would process all these resources in the namespace and execute the `ReadAndWrite` ones first (in serial: one at a time to avoid write conflicts) and then the `ReadOnly` ones (in parallel). +The _secureCodeBox operator_ would process all these CRDs in the namespace and execute the *read and write* ones first in serial only one at a time to avoid write conflicts and then the *read only* ones in parallel. +//// +#30: What namespace is meant here? +//// [source,yaml] ---- @@ -95,14 +98,14 @@ The Execution Flow would then look something like this: ====== Pros -* Only one Implementation -* Pretty Generic to expand and test out new ideas without having to modify the secureCodeBox Operator +* Only one implementation. +* Pretty generic to expand and test out new ideas without having to modify the _secureCodeBox operator_. ====== Cons -* Possible "over-abstraction"? -* Need to refactor the ElasticSearch PersistenceProvider -* The "General Implementation" will be harder than the individual ones +* Possibly an "`over-abstraction`". +* Need to refactor the _persistence-elastic_ provider. +* The "`general implementation`" will be harder than the individual ones. ===== Solution Approach 1: Keep Split between Persistence Provider and MetaData Provider From 6584335354bf61062808c22bf9f189fa3a71e251 Mon Sep 17 00:00:00 2001 From: Sven Strittmatter Date: Mon, 29 Jun 2020 12:45:14 +0200 Subject: [PATCH 11/17] #30 Adapt solution 2 to prvious paragraphs Signed-off-by: Sven Strittmatter --- docs/adr/adr_0001.adoc | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/adr/adr_0001.adoc b/docs/adr/adr_0001.adoc index 900fc6ec..80f4e774 100644 --- a/docs/adr/adr_0001.adoc +++ b/docs/adr/adr_0001.adoc @@ -107,9 +107,9 @@ The Execution Flow would then look something like this: * Need to refactor the _persistence-elastic_ provider. * The "`general implementation`" will be harder than the individual ones. -===== Solution Approach 1: Keep Split between Persistence Provider and MetaData Provider +===== Solution Approach 2: Keep Split between Persistence Provider and MetaData Provider -Keep PersistenceProvider as they are and introduce new "MetaDataProvider" CRD which gets executed before the PersistenceProviders by the operator. +Keep _PersistenceProvider_ as they are and introduce new _MetaDataProvider_ CRD which gets executed before the _PersistenceProviders_ by the __secureCodeBox operator_. .... ┌ Persistence Provider─ ─ ─ ─ ─ ─ ─ ─ @@ -124,12 +124,12 @@ Keep PersistenceProvider as they are and introduce new "MetaDataProvider" CRD wh ====== Pros -* Quicker to implement -* Might be worth it to have a separate concept for it +* Quicker to implement. +* Might be worth it to have a separate concept for it. ====== Cons -* Really worth introducing a new CRD for everything, especially when the are conceptually pretty close? +* Not sure if it worth to introduce a new CRD for everything, especially when it's conceptually pretty close to to something already existing. ==== Question 2: How Should the Execution Model Look like for Each? From 66da9b41d9ba3045ebb5defba579f8f22132699c Mon Sep 17 00:00:00 2001 From: Sven Strittmatter Date: Mon, 29 Jun 2020 13:26:22 +0200 Subject: [PATCH 12/17] #30 Reqork the question 2 part Signed-off-by: Sven Strittmatter --- docs/adr/adr_0001.adoc | 48 +++++++++++++++++++++--------------------- 1 file changed, 24 insertions(+), 24 deletions(-) diff --git a/docs/adr/adr_0001.adoc b/docs/adr/adr_0001.adoc index 80f4e774..47067061 100644 --- a/docs/adr/adr_0001.adoc +++ b/docs/adr/adr_0001.adoc @@ -131,14 +131,14 @@ Keep _PersistenceProvider_ as they are and introduce new _MetaDataProvider_ CRD * Not sure if it worth to introduce a new CRD for everything, especially when it's conceptually pretty close to to something already existing. -==== Question 2: How Should the Execution Model Look like for Each? +==== Question 2: How Should the Execution Model Look like for Each Concept? ===== Solution Approach 1: Like the Persistence Provider -Basically a docker container which process takes two command line args: +Basically a docker container which process findings takes two arguments: -* A pre-signed URL to download the findings from -* A pre-signed URL to upload the modified findings to +. A pre-defined URL to download the findings from. +. A pre-defined URL to upload the modified findings to. Examples: @@ -148,36 +148,36 @@ Examples: ====== Pros -* on liner with the current implementations -* code overhead / wrapper code is pretty minimal -* zero scale - no resource costs when nothing is running +* One liner with the current implementations. +* Code overhead / wrapper code is pretty minimal. +* Zero scale: no resource costs when nothing is running. ===== Cons -* results in too many k8s jobs? -** resource blocking on finished resources -** ttlAfterFinished enabled -* container runtime overhead (especially time) +* May results in too many Kubernetes jobs. +** Resource blocking on finished resources. +** `ttlAfterFinished` enabled. +* Container runtime overhead (especially time). -=== Option 2: A WebHooks Like Concept +===== Solution Approach 2: A WebHooks Like Concept -Analog to kubernetes webhooks. Https server receiving findings and returning results. +Analog to kubernetes webhooks: HTTP server receiving findings and returning results. -==== Pros +===== Pros -* MilliSeconds instead of seconds for processing -* No ContainerCreation Overhead -* No additional k8s jobs needed +* Milliseconds instead of seconds for processing. +* No overhead for container Creation. +* No additional kubernetes jobs needed. ===== Cons -* Introduces new running Services that need to be maintained and have uptime -* Code Overhead / Boilerplate (Can be mitigated by SDK) -* Debugging of individual MetaDataProvider is harder as everything is handled by a single service -* Introduces "New" Concept -* Certificate Management for webhook services (`cert-manager` required by default?) -* Scaling for systems with lots of load could be a problem -* One service per namespace (multiple tenants) needed => results in many running active services which is resource consuming +* Introduces new running services which needs to be maintained and have uptime. +* Code overhead / boilerplate (Can be mitigated by an SDK). +* Debugging of individual _MetaDataProvider_ is harder than a single service which handles everything. +* Introduces "`new`"cConcept. +* Certificate management for webhook services (`cert-manager` required by default?). +* Scaling for systems with lots of load could be a problem. +* One service per namespace (multiple tenants) needed -> results in many running active services which is resource consuming. == Decision From 6e42744af6cc22825ccbc93a28767b292c15d531 Mon Sep 17 00:00:00 2001 From: Sven Strittmatter Date: Mon, 29 Jun 2020 14:13:32 +0200 Subject: [PATCH 13/17] #30 Rework the decission part Signed-off-by: Sven Strittmatter --- docs/adr/adr_0001.adoc | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/adr/adr_0001.adoc b/docs/adr/adr_0001.adoc index 47067061..71e59eef 100644 --- a/docs/adr/adr_0001.adoc +++ b/docs/adr/adr_0001.adoc @@ -181,17 +181,18 @@ Analog to kubernetes webhooks: HTTP server receiving findings and returning resu == Decision -Regarding the Question 1 it seems that both solution approaches are resulting in the same execution model. We decided to implement solution approach 1 and unify both concepts into a more general concept with the name _"hook concept"_. Therefore we exchange the existing name `persistenceProvider` for phase 3 in the execution model with a more general term `DataProcessing`: +Regarding question 1 it seems that both solution approaches are resulting in the same execution model. We decided to implement solution approach 1 and unify both concepts into a more general concept with the name _hook concept_. Therefore we exchange the existing name _PersistenceProvider_ for phase 3 in the execution model with a more general term _DataProcessing_: +// #30 I suggest the term "processing" instead of data processing/ .... ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ -│ Scanning ├─────────▶│ Parsing ├─────────▶│ DataProcessing │ +│ scanning ├─────────▶│ parsing ├─────────▶│ data processing │ │ (Phase 1) │ │ (Phase 2) │ │ (Phase 3) │ └──────────────────┘ └──────────────────┘ └──────────────────┘ .... -Regarding the Question 2 we decided to implement the solution approach 1 with a job-based approach (no active service component needed). -The Phase 3 `DataProcessing` will be therefore split into to separate phases named `ReadAndWriteHooks (3.1)` and `ReadOnlyHooks (3.2)` +Regarding question 2 we decided to implement the solution approach 1 with a job-based approach (no active service component needed). Therefore the phase 3 _data processing_ will be split into two separate phases named _ReadAndWriteHooks_ (3.1) and _ReadOnlyHooks_ (3.2) +// #30 to what refers 3.1 and 3.2? .... ┌ DataProcessing: ReadOnlyHooks ─ ─ ─ From fd54f578594371e78e41b42be06e75012c36f484 Mon Sep 17 00:00:00 2001 From: Sven Strittmatter Date: Mon, 29 Jun 2020 14:18:14 +0200 Subject: [PATCH 14/17] #30 Rework the consequences part Signed-off-by: Sven Strittmatter --- docs/adr/adr_0001.adoc | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/adr/adr_0001.adoc b/docs/adr/adr_0001.adoc index 71e59eef..d2f7c4c7 100644 --- a/docs/adr/adr_0001.adoc +++ b/docs/adr/adr_0001.adoc @@ -207,12 +207,12 @@ Regarding question 2 we decided to implement the solution approach 1 with a job- == Consequences -With the new `Hook Concept` we open the `DataProcessing` Phase 3 to a more intuitive and flexible architecture. It is easier to understand because _WebHooks_ are already a well known concept. It is possible to keep the existing implementation of the `persistenceProviders` and to integrate them with a lot of other possible data processing components in a more general fashion. In the end, this step will result in a lot of additional feature possibilities, which go fare beyond the existing ones. Therefore we only need to implement this concept once in the secureCodeBox Operator and new ideas for extending the DataProcessing will not enforce conceptual or architectural changes. +With the new _hook concept_ we open the _phase 3 data processing_ to a more intuitive and flexible architecture. It is easier to understand because _WebHooks_ are already a well known concept. It is possible to keep the existing implementation of the _PersistenceProvider_ and integrate them with a lot of other possible data processing components in a more general fashion. In the end, this step will result in a lot of additional feature possibilities, which go far beyond the existing ones proposed here. Therefore we only need to implement this concept once in the _secureCodeBox operator_ and new ideas for extending the _DataProcessing_ will not enforce conceptual or architectural changes. Ideas for additional data processing hooks: -* Notifier-Hooks (ReadOnlyHook) e.g., for chat (slack, teams etc.), metric, alerting systems -* MetaData enrichment hooks (ReadAndWriteHook) -* FilterData Hooks (e.g., false/positive handling) (ReadAndWriteHook) -* SystemIntegration Hooks (ReadOnlyHook) e.g., for ticketing systems like Jira -* CascadingScans Hooks (ReadOnlyHook) e.g., for starting new security scans based on findings +* Notifier hooks (_ReadOnlyHook_) e.g., for chat (slack, teams etc.), metric, alerting systems +* MetaData enrichment hooks (_ReadAndWriteHook_) +* FilterData hooks (_ReadAndWriteHook_) (e.g., false/positive handling) +* SystemIntegration hooks (_ReadOnlyHook_) e.g., for ticketing systems like Jira +* CascadingScans hooks (_ReadOnlyHook_) e.g., for starting new security scans based on findings From bbf99738ef6bf6b2f9bfbd5c2028881fc0148dcd Mon Sep 17 00:00:00 2001 From: Sven Strittmatter Date: Tue, 30 Jun 2020 16:43:50 +0200 Subject: [PATCH 15/17] #30 Simplify name ofphase three as sugested Signed-off-by: Sven Strittmatter --- docs/adr/adr_0001.adoc | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/docs/adr/adr_0001.adoc b/docs/adr/adr_0001.adoc index d2f7c4c7..b668c05a 100644 --- a/docs/adr/adr_0001.adoc +++ b/docs/adr/adr_0001.adoc @@ -181,17 +181,16 @@ Analog to kubernetes webhooks: HTTP server receiving findings and returning resu == Decision -Regarding question 1 it seems that both solution approaches are resulting in the same execution model. We decided to implement solution approach 1 and unify both concepts into a more general concept with the name _hook concept_. Therefore we exchange the existing name _PersistenceProvider_ for phase 3 in the execution model with a more general term _DataProcessing_: +Regarding question 1 it seems that both solution approaches are resulting in the same execution model. We decided to implement solution approach 1 and unify both concepts into a more general concept with the name _hook concept_. Therefore we exchange the existing name _PersistenceProvider_ for phase 3 in the execution model with a more general term _processing_: -// #30 I suggest the term "processing" instead of data processing/ .... ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ -│ scanning ├─────────▶│ parsing ├─────────▶│ data processing │ +│ scanning ├─────────▶│ parsing ├─────────▶│ processing │ │ (Phase 1) │ │ (Phase 2) │ │ (Phase 3) │ └──────────────────┘ └──────────────────┘ └──────────────────┘ .... -Regarding question 2 we decided to implement the solution approach 1 with a job-based approach (no active service component needed). Therefore the phase 3 _data processing_ will be split into two separate phases named _ReadAndWriteHooks_ (3.1) and _ReadOnlyHooks_ (3.2) +Regarding question 2 we decided to implement the solution approach 1 with a job-based approach (no active service component needed). Therefore the phase 3 _processing_ will be split into two separate phases named _ReadAndWriteHooks_ (3.1) and _ReadOnlyHooks_ (3.2) // #30 to what refers 3.1 and 3.2? .... @@ -207,9 +206,9 @@ Regarding question 2 we decided to implement the solution approach 1 with a job- == Consequences -With the new _hook concept_ we open the _phase 3 data processing_ to a more intuitive and flexible architecture. It is easier to understand because _WebHooks_ are already a well known concept. It is possible to keep the existing implementation of the _PersistenceProvider_ and integrate them with a lot of other possible data processing components in a more general fashion. In the end, this step will result in a lot of additional feature possibilities, which go far beyond the existing ones proposed here. Therefore we only need to implement this concept once in the _secureCodeBox operator_ and new ideas for extending the _DataProcessing_ will not enforce conceptual or architectural changes. +With the new _hook concept_ we open the _phase 3 processing_ to a more intuitive and flexible architecture. It is easier to understand because _WebHooks_ are already a well known concept. It is possible to keep the existing implementation of the _PersistenceProvider_ and integrate them with a lot of other possible processing components in a more general fashion. In the end, this step will result in a lot of additional feature possibilities, which go far beyond the existing ones proposed here. Therefore we only need to implement this concept once in the _secureCodeBox operator_ and new ideas for extending the _DataProcessing_ will not enforce conceptual or architectural changes. -Ideas for additional data processing hooks: +Ideas for additional processing hooks: * Notifier hooks (_ReadOnlyHook_) e.g., for chat (slack, teams etc.), metric, alerting systems * MetaData enrichment hooks (_ReadAndWriteHook_) From 5c69bd53406bfc4eef43d1fc1284f72b10996211 Mon Sep 17 00:00:00 2001 From: Sven Strittmatter Date: Tue, 30 Jun 2020 16:46:18 +0200 Subject: [PATCH 16/17] #30 Add missing numbers at images Signed-off-by: Sven Strittmatter --- docs/adr/adr_0001.adoc | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/adr/adr_0001.adoc b/docs/adr/adr_0001.adoc index b668c05a..418a3bac 100644 --- a/docs/adr/adr_0001.adoc +++ b/docs/adr/adr_0001.adoc @@ -185,7 +185,7 @@ Regarding question 1 it seems that both solution approaches are resulting in the .... ┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ -│ scanning ├─────────▶│ parsing ├─────────▶│ processing │ +│ scanning ├─────────▶│ parsing ├─────────▶│ processing │ │ (Phase 1) │ │ (Phase 2) │ │ (Phase 3) │ └──────────────────┘ └──────────────────┘ └──────────────────┘ .... @@ -194,11 +194,11 @@ Regarding question 2 we decided to implement the solution approach 1 with a job- // #30 to what refers 3.1 and 3.2? .... - ┌ DataProcessing: ReadOnlyHooks ─ ─ ─ - ┌ DataProcessing: ReadAndWriteHooks ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┌────────────────────────────────┐ │ + ┌ 3.2 processing: ReadOnlyHooks ─ ─ ─ + ┌ 3.1 processing: ReadAndWriteHooks ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┌────────────────────────────────┐ │ ┌───────────────────────┐ │ ┌──┼▶│ Elastic PersistenceProvider │ ┌──────────────────┐ ┌──────────────────┐ │ │ ReadAndWrite Hook #1 │ ┌───────────────────────┐ │ └────────────────────────────────┘ │ -│ Scan ├──▶│ Parsing │────▶│ "MyMetaDataProvider" ├─▶│ ReadAndWrite Hook #2 │─┼──┤ │ ┌────────────────────────────────┐ +│ scanning ├──▶│ parsing │────▶│ "MyMetaDataProvider" ├─▶│ ReadAndWrite Hook #2 │─┼──┤ │ ┌────────────────────────────────┐ └──────────────────┘ └──────────────────┘ │ └───────────────────────┘ └───────────────────────┘ └───▶│ DefectDojo PersistenceProvider │ │ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ │ └────────────────────────────────┘ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ From 74b2c60414e15c8fbce47b56481de4f8035d2b02 Mon Sep 17 00:00:00 2001 From: Sven Strittmatter Date: Wed, 1 Jul 2020 12:36:10 +0200 Subject: [PATCH 17/17] #30 Fixed the unknown namespace error in ADR-0001 Signed-off-by: Sven Strittmatter --- docs/adr/adr_0001.adoc | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/docs/adr/adr_0001.adoc b/docs/adr/adr_0001.adoc index 418a3bac..87d68733 100644 --- a/docs/adr/adr_0001.adoc +++ b/docs/adr/adr_0001.adoc @@ -66,10 +66,7 @@ We could do a similar thing and introduce CRD which allows to execute "`custom c These could be implemented with a `type` attribute, which declares if they are *read only* or *read and write*. -The _secureCodeBox operator_ would process all these CRDs in the namespace and execute the *read and write* ones first in serial only one at a time to avoid write conflicts and then the *read only* ones in parallel. -//// -#30: What namespace is meant here? -//// +The _secureCodeBox operator_ would process all these CRDs in the namespace of the scan and execute the *read and write* ones first in serial only one at a time to avoid write conflicts and then the *read only* ones in parallel. [source,yaml] ----