From 71c5212465f6ab40c3d0946f80826e4dbd72dc0c Mon Sep 17 00:00:00 2001 From: Daniel Patanin Date: Tue, 29 Sep 2020 09:14:48 +0200 Subject: [PATCH 1/4] Move adr of securecodebox.io to main repo Since we changed the framework for our website the decision described in the respective adr file will be archived or forgotten about. Furthermore this website is a tool we use for our main project, thus the decision about what "tool" we use for documentation should be saved in the main repository. --- docs/adr/adr_0001.adoc | 253 ++++++++------------------------ docs/adr/adr_0002.adoc | 318 +++++++++++++++++++++++------------------ docs/adr/adr_0003.adoc | 182 +++++++++++++++++++++++ 3 files changed, 415 insertions(+), 338 deletions(-) create mode 100644 docs/adr/adr_0003.adoc diff --git a/docs/adr/adr_0001.adoc b/docs/adr/adr_0001.adoc index 87d68733..53e4e95d 100644 --- a/docs/adr/adr_0001.adoc +++ b/docs/adr/adr_0001.adoc @@ -1,214 +1,77 @@ -[[ADR-0000]] -= ADR-0000: How can we introduce a more general extension concept for data processing modules? +[[ADR-0001]] += ADR-0001: Choosing the framework for the new secureCodeBox Website [cols="h,d",grid=rows,frame=none,stripes=none,caption="Status",%autowidth] |==== - +// Use one of the ADR status parameter based on status +// Please add a cross reference link to the new ADR on 'superseded' ADR. +// e.g.: {adr_suposed_by} <> | Status | ACCEPTED | Date -| 2020-05-20 +| 2019-08-21 | Author(s) -| Jannik Hollenbach , - Jorge Estigarribia , - Robert Seedorff , - Sven Strittmatter +| Daniel Patanin daniel.patanin@iteratec.com, + Jannick Hollenbach jannick.hollenbach@iteratec.com +// ... |==== == Context -=== Status Quo - -One major challenge implementing the _secureCodeBox_ is to provide a flexible and modular architecture, which enables the open source community to easily understand the concepts and especially to extend the _secureCodeBox_ with individual features. Therefore we decided to separate the process stages of a single security scan (instance of _scanType_ custom resource definition; further abbreviated with _CRD_) in three major phases: - -.... -┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ -│ scanning ├─────────▶│ parsing ├─────────▶│ persisting │ -│ (phase 1) │ │ (phase 2) │ │ (phase 3) │ -└──────────────────┘ └──────────────────┘ └──────────────────┘ -.... - -By now the phase 3 "`persisting`" was implemented by so called _PersistenceProviders_ (e.g., the _persistence-elastic_ provider which is responsible for persisting all findings in a given elasticsearch database). The _secureCodeBox_ Operator is aware of this 3 phases and is responsible for the state model and execution of each security scan. - -=== Problem and Question - -We identified different additional use cases with a more "`data processing oriented`" pattern than the implemented phase 3 "`persisting`" indicates. For example, we implemented a so called _MetaDataProvider_ feature, which is responsible for enhancing each security finding with additional metadata. But the _MetaDataProvider_ must be executed after the phase 2 "`parsing`" and before the phase 3 "`persisting`" because it depends on the parsed finding results (which will be enhanced) and the updated findings should be also persisted. - -To find a proper solution, we split the topic into the following two questions: - -. Should we unify the concepts _MetaDataProvider_ and _PersistenceProvider_? -. How should the execution model look like for each concept? - -==== Question 1: Should We Unify the Concepts MetaDataProvider and PersistenceProvider? - -===== Solution Approach 1: Unify - -Both "`modules`" are "`processing`" the security findings, which were generated in the phase 2 "`parsing`", -but there is one major difference between them: - -* a _PersistenceProvider_ is processing the findings *read only*, and -* a _MetaDataProvider_ is processing the findings *read and write*. - -There is a similar concept in Kubernetes called https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/[AdmissionController], but with the exception that the will be executed before a resource is created. - -There are two variants of _AdmissionControllers_: - -. _ValidatingWebhookConfiguration_: *read only*, *executed last*; and -. _MutatingWebhookConfiguration_: *read and write*, *executed first*. - -We could do a similar thing and introduce CRD which allows to execute "`custom code`" (depends on the second question) after a scan has completed (meaning both phases "`scan`" and "`parsing`" were done). Some name ideas: - -* _ScanHooks_ -* _ScanCompletionHooks_ -* _FindingProcessors_ - -These could be implemented with a `type` attribute, which declares if they are *read only* or *read and write*. - -The _secureCodeBox operator_ would process all these CRDs in the namespace of the scan and execute the *read and write* ones first in serial only one at a time to avoid write conflicts and then the *read only* ones in parallel. - -[source,yaml] ----- -apiVersion: execution.experimental.securecodebox.io/v1 -kind: ScanCompletionHook -metadata: - name: my-metadata -spec: - type: ReadAndWrite - # If implemented like the current persistence provider - image: my-metadata:v2.0.0 ----- - -The Execution Flow would then look something like this: - -.... - ┌ ReadOnly─Hooks─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ - ┌ ReadAndWriteHooks ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┌────────────────────────────────┐ │ - ┌───────────────────────┐ │ ┌──┼▶│ Elastic PersistenceProvider │ -┌──────────────────┐ ┌──────────────────┐ │ │ ReadAndWrite Hook #1 │ ┌───────────────────────┐ │ └────────────────────────────────┘ │ -│ Scan ├──▶│ Parsing │────▶│ "MyMetaDataProvider" ├─▶│ ReadAndWrite Hook #2 │─┼──┤ │ ┌────────────────────────────────┐ -└──────────────────┘ └──────────────────┘ │ └───────────────────────┘ └───────────────────────┘ └───▶│ DefectDojo PersistenceProvider │ │ - ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ │ └────────────────────────────────┘ - ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ -.... - -====== Pros - -* Only one implementation. -* Pretty generic to expand and test out new ideas without having to modify the _secureCodeBox operator_. - -====== Cons - -* Possibly an "`over-abstraction`". -* Need to refactor the _persistence-elastic_ provider. -* The "`general implementation`" will be harder than the individual ones. - -===== Solution Approach 2: Keep Split between Persistence Provider and MetaData Provider - -Keep _PersistenceProvider_ as they are and introduce new _MetaDataProvider_ CRD which gets executed before the _PersistenceProviders_ by the __secureCodeBox operator_. - -.... - ┌ Persistence Provider─ ─ ─ ─ ─ ─ ─ ─ - ┌ MetaData Provider ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┌────────────────────────────────┐ │ - ┌───────────────────────┐ │ ┌──┼▶│ Elastic PersistenceProvider │ -┌──────────────────┐ ┌──────────────────┐ │ │ ReadAndWrite Hook #1 │ ┌───────────────────────┐ │ └────────────────────────────────┘ │ -│ Scan ├──▶│ Parsing │────▶│ "MyMetaDataProvider" ├─▶│ ReadAndWrite Hook #2 │─┼──┤ │ ┌────────────────────────────────┐ -└──────────────────┘ └──────────────────┘ │ └───────────────────────┘ └───────────────────────┘ └───▶│ DefectDojo PersistenceProvider │ │ - ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ │ └────────────────────────────────┘ - ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ -.... - -====== Pros - -* Quicker to implement. -* Might be worth it to have a separate concept for it. - -====== Cons - -* Not sure if it worth to introduce a new CRD for everything, especially when it's conceptually pretty close to to something already existing. - -==== Question 2: How Should the Execution Model Look like for Each Concept? - -===== Solution Approach 1: Like the Persistence Provider - -Basically a docker container which process findings takes two arguments: - -. A pre-defined URL to download the findings from. -. A pre-defined URL to upload the modified findings to. - -Examples: - -* NodeJS: `node my-metadata.js "https://storage.googleapi.com/..." "https://storage.googleapi.com/..."` -* Java: `java my-metadata.jar "https://storage.googleapi.com/..." "https://storage.googleapi.com/..."` -* Golang: `./my-metadata "https://storage.googleapi.com/..." "https://storage.googleapi.com/..."` - -====== Pros - -* One liner with the current implementations. -* Code overhead / wrapper code is pretty minimal. -* Zero scale: no resource costs when nothing is running. - -===== Cons - -* May results in too many Kubernetes jobs. -** Resource blocking on finished resources. -** `ttlAfterFinished` enabled. -* Container runtime overhead (especially time). - -===== Solution Approach 2: A WebHooks Like Concept - -Analog to kubernetes webhooks: HTTP server receiving findings and returning results. - -===== Pros - -* Milliseconds instead of seconds for processing. -* No overhead for container Creation. -* No additional kubernetes jobs needed. - -===== Cons - -* Introduces new running services which needs to be maintained and have uptime. -* Code overhead / boilerplate (Can be mitigated by an SDK). -* Debugging of individual _MetaDataProvider_ is harder than a single service which handles everything. -* Introduces "`new`"cConcept. -* Certificate management for webhook services (`cert-manager` required by default?). -* Scaling for systems with lots of load could be a problem. -* One service per namespace (multiple tenants) needed -> results in many running active services which is resource consuming. +There are tons of different frameworks for building websites out there. We must choose the most fitting one for our use, fulfilling our mandatory requirements: + +• Common programming language, if applicable easy to learn +• Overall easy to use and start-up, also locally +• Tutorials, examples and a good documentation +• Bonus points for great and many easy to use templates and plugins +• Needs continuous support and contribution +• Must be able to be deployed as GitHub pages + +We will choose from the following popular/trending: + +https://gridsome.org/[Gridsome] + +https://www.gatsbyjs.org/[Gatsby] + +https://gohugo.io/[Hugo] + +https://jekyllrb.com/[Jekyll] + +=== Research + +These frameworks do all fulfill the requirements to the extent that I estimate them as wellsuited. First, I researched the listed features on the respective sites or quickly googled after it +specifically and found instantly the requested feature. I followed up with a general overview +of how old the frameworks, how popular they are and for example pages build with them. +Afterwards I searched for comparison blogs and posts, mostly to examine their comments. +Most of these „pro-cons “-posts are inaccurate and very superficial, but luckily because of that +the comment sections hold interesting discussions and comparisons from overall features and +usability to specific issues and problems of each framework and which framework fits what +use-cases in general. After this research I’ve come to a majority of similar experience sharing +and discussions. These described the distribution of these frameworks as follows (roughly +summarized): + +Gridsome is like Gatsby just for VueJS. +Gatsby is blazing fast after building the pages but requires a little bit more understanding of +JavaScript and React and may not be as easy to get behind if you’ve never built a site with a +static site generator before. +Hugo is fast in building and based on Golang. But as a newbie to that language you’ll find yourself using the documentation very much, unless you learn this language to a curtain depth. +Jekyll is simple in templating and very good for quickly starting a small blog site but based on +ruby and therefore requires ruby dependencies. == Decision -Regarding question 1 it seems that both solution approaches are resulting in the same execution model. We decided to implement solution approach 1 and unify both concepts into a more general concept with the name _hook concept_. Therefore we exchange the existing name _PersistenceProvider_ for phase 3 in the execution model with a more general term _processing_: - -.... -┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ -│ scanning ├─────────▶│ parsing ├─────────▶│ processing │ -│ (Phase 1) │ │ (Phase 2) │ │ (Phase 3) │ -└──────────────────┘ └──────────────────┘ └──────────────────┘ -.... - -Regarding question 2 we decided to implement the solution approach 1 with a job-based approach (no active service component needed). Therefore the phase 3 _processing_ will be split into two separate phases named _ReadAndWriteHooks_ (3.1) and _ReadOnlyHooks_ (3.2) -// #30 to what refers 3.1 and 3.2? - -.... - ┌ 3.2 processing: ReadOnlyHooks ─ ─ ─ - ┌ 3.1 processing: ReadAndWriteHooks ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┌────────────────────────────────┐ │ - ┌───────────────────────┐ │ ┌──┼▶│ Elastic PersistenceProvider │ -┌──────────────────┐ ┌──────────────────┐ │ │ ReadAndWrite Hook #1 │ ┌───────────────────────┐ │ └────────────────────────────────┘ │ -│ scanning ├──▶│ parsing │────▶│ "MyMetaDataProvider" ├─▶│ ReadAndWrite Hook #2 │─┼──┤ │ ┌────────────────────────────────┐ -└──────────────────┘ └──────────────────┘ │ └───────────────────────┘ └───────────────────────┘ └───▶│ DefectDojo PersistenceProvider │ │ - ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ │ └────────────────────────────────┘ - ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ -.... +So, it seems that Hugo is a pretty good choice for sites with many, many…. like many pages. +Jekyll seems to fit for a quick build. Gatsby and Gridsome require a bit more time to learn but +have their advantages in speed and growth of the site. And whether you choose Gridsome over +Gatsby relies on whether you want to use VueJS or not. -== Consequences - -With the new _hook concept_ we open the _phase 3 processing_ to a more intuitive and flexible architecture. It is easier to understand because _WebHooks_ are already a well known concept. It is possible to keep the existing implementation of the _PersistenceProvider_ and integrate them with a lot of other possible processing components in a more general fashion. In the end, this step will result in a lot of additional feature possibilities, which go far beyond the existing ones proposed here. Therefore we only need to implement this concept once in the _secureCodeBox operator_ and new ideas for extending the _DataProcessing_ will not enforce conceptual or architectural changes. +Finally we’ve decided to use Gatsby. Some of the main reasons is it’s fast performance, the extensive documentation and tutorials and also the language, since Hugo (the +other framework we considered mainly) is based on Golang, and as for my part as a developer I +feel completely comfortable and prefer working with JSX. Overall it comes down to preferences mostly, since we’re not going to build a giant Website, nor are we planning on implementing “crazy” Features. -Ideas for additional processing hooks: +== Consequences -* Notifier hooks (_ReadOnlyHook_) e.g., for chat (slack, teams etc.), metric, alerting systems -* MetaData enrichment hooks (_ReadAndWriteHook_) -* FilterData hooks (_ReadAndWriteHook_) (e.g., false/positive handling) -* SystemIntegration hooks (_ReadOnlyHook_) e.g., for ticketing systems like Jira -* CascadingScans hooks (_ReadOnlyHook_) e.g., for starting new security scans based on findings +For the integration of our multi-repository documentation we’ll use +Antora if working this out with Gatsby is going to be more difficult than integrating Antora. +We’re aware that using Gatsby requires a bit more maintenance and has the drawback, that if +anybody else will maintain or work on the website, this person will need to at least understand +the basics of React and GraphQL. \ No newline at end of file diff --git a/docs/adr/adr_0002.adoc b/docs/adr/adr_0002.adoc index 7069f35a..60f6eba1 100644 --- a/docs/adr/adr_0002.adoc +++ b/docs/adr/adr_0002.adoc @@ -1,5 +1,5 @@ [[ADR-0002]] -= ADR-0002: How can we introduce a mechanism to start specialized scans on the results of previous scans? += ADR-0000: How can we introduce a more general extension concept for data processing modules? [cols="h,d",grid=rows,frame=none,stripes=none,caption="Status",%autowidth] |==== @@ -12,6 +12,7 @@ | Author(s) | Jannik Hollenbach , + Jorge Estigarribia , Robert Seedorff , Sven Strittmatter |==== @@ -20,163 +21,194 @@ === Status Quo -Currently scans by the secureCodeBox are single focused on a specific tool. -Combining multiple scans requires manual or scripting by the user to use the results of a scan (e.g. Nmap) as a input for another scanner (e.g. SSLyze) +One major challenge implementing the _secureCodeBox_ is to provide a flexible and modular architecture, which enables the open source community to easily understand the concepts and especially to extend the _secureCodeBox_ with individual features. Therefore we decided to separate the process stages of a single security scan (instance of _scanType_ custom resource definition; further abbreviated with _CRD_) in three major phases: + +.... +┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ +│ scanning ├─────────▶│ parsing ├─────────▶│ persisting │ +│ (phase 1) │ │ (phase 2) │ │ (phase 3) │ +└──────────────────┘ └──────────────────┘ └──────────────────┘ +.... + +By now the phase 3 "`persisting`" was implemented by so called _PersistenceProviders_ (e.g., the _persistence-elastic_ provider which is responsible for persisting all findings in a given elasticsearch database). The _secureCodeBox_ Operator is aware of this 3 phases and is responsible for the state model and execution of each security scan. === Problem and Question -How can the results of a scan be used to automatically configure subsequent specialized scans for identified targets. +We identified different additional use cases with a more "`data processing oriented`" pattern than the implemented phase 3 "`persisting`" indicates. For example, we implemented a so called _MetaDataProvider_ feature, which is responsible for enhancing each security finding with additional metadata. But the _MetaDataProvider_ must be executed after the phase 2 "`parsing`" and before the phase 3 "`persisting`" because it depends on the parsed finding results (which will be enhanced) and the updated findings should be also persisted. -In general we want to describe cascading scans like: +To find a proper solution, we split the topic into the following two questions: -``` -+--------+ +--------+ +--------+ -| scan 1 |-- result -->| scan 2 |-- result -->| scan 3 | -+--------+ +--------+ | +--------+ - | - | +--------+ - +---->| scan 4 | - +--------+ -```` +. Should we unify the concepts _MetaDataProvider_ and _PersistenceProvider_? +. How should the execution model look like for each concept? -A concrete example: +==== Question 1: Should We Unify the Concepts MetaDataProvider and PersistenceProvider? -``` -+----------------+ +-----------------+ +-----------+ -| <> | | <> | | <> | -| find all hosts |-- IP -->| find open ports |-- port 443 -->| check TLS | -+----------------+ +-----------------+ | +-----------+ - | - | +-------------+ - | | <> | - +------->| check HTTPd | - +-------------+ -``` +===== Solution Approach 1: Unify -The solution should fulfill the following criteria: +Both "`modules`" are "`processing`" the security findings, which were generated in the phase 2 "`parsing`", +but there is one major difference between them: -- The "rules" used to describe which subsequent scans can be executed should be modular, so that they can be packaged together with the scan types. -- It should be possible for a user to select which scan rules should be applied -- Protections should be in place to ensure that the clusters are not completely overwhelmed by these automatically created scans. Especially circular structures which create a infinite number of scans should be prevented. +* a _PersistenceProvider_ is processing the findings *read only*, and +* a _MetaDataProvider_ is processing the findings *read and write*. -== Decision +There is a similar concept in Kubernetes called https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/[AdmissionController], but with the exception that the will be executed before a resource is created. -It was decided to implement these rules as Custom Resource Definitions (CRDs) in Kubernetes. -This allows the Helm Charts of the scanners to package related rules for the scanner together with their ScanTypes. +There are two variants of _AdmissionControllers_: -=== Defining CascadingRule +. _ValidatingWebhookConfiguration_: *read only*, *executed last*; and +. _MutatingWebhookConfiguration_: *read and write*, *executed first*. -The so called "CascadingRules" consist of a "matches" section which contains one or multiple rules which are compared against findings. -When a finding matches a rule the "scanSpec" section will then be used to create a new scan. -To customize the scan to match the finding, the [mustache](https://github.com/janl/mustache.js) templating language can be used to reference fields of the finding. +We could do a similar thing and introduce CRD which allows to execute "`custom code`" (depends on the second question) after a scan has completed (meaning both phases "`scan`" and "`parsing`" were done). Some name ideas: -```yaml -apiVersion: "cascading.experimental.securecodebox.io/v1" -kind: CascadingRule -metadata: - name: "tls-scans" - labels: - # Described how "invasive" the scan is. - # Possible values: "invasive" or "non-invasive" - # CascadingRules are considered "invasive" when the Scan they start actively sends out packages with attack payloads. - securecodebox.io/invasive: non-invasive - # Described the intensiveness level on a scanning and computational resource level. - # Possible values: "ligh", "medium", "intense" - # CascadingRules are considered more "intensive" when the Scan they start consumes lots of computational resources like RAM, CPU, or Network - securecodebox.io/intensive: light -spec: - matches: - # CascadingRule triggers if a finding matches at least one of the anyOf matchers - # With the first version of this implementation only anyOf would be supported. - # If this turns out to be lacking and other operators (like `allOf` can be introduced without breaking changes) - anyOf: - # define an explicit "port" as finding and a given port number - - category: "Open Port" - attributes: - port: 443 - service: "https" - # define an "port service" finding (any port) - - category: "Open Port" - attributes: - service: "https" - scanSpec: - name: "sslyze" - parameters: ["--regular", "{{attributes.hostname}}"] -``` - -=== Using CascadingRules - -By default no cascading Rules will be used. - -```yaml -# Nmap Scan without cascading rules -apiVersion: "execution.experimental.securecodebox.io/v1" -kind: Scan -metadata: - name: "portscan-berlin-wifi" - label: - office: berlin - vlan: wifi -spec: - name: "nmap" - parameters: ["-sV", "10.42.0.0/16"] -``` +* _ScanHooks_ +* _ScanCompletionHooks_ +* _FindingProcessors_ -To enable cascading rules you need to specify a label selector to select the cascading rules you'd like +These could be implemented with a `type` attribute, which declares if they are *read only* or *read and write*. -```yaml -apiVersion: "execution.experimental.securecodebox.io/v1" -kind: Scan -metadata: - name: "portscan-berlin-wifi" - label: - office: berlin - vlan: wifi -spec: - cascades: - matchLabels: - # Uses all CascadingRules in the namespace which are labelled as "non-invasive" and a intensiveness level of "light" - securecodebox.io/invasive: non-invasive - securecodebox.io/intensive: light - name: "nmap" - parameters: ["-sV", "10.42.0.0/16"] -``` - -To implicitly enable all cascading rules (not-recommended) a empty label selector can be used - -```yaml -apiVersion: "execution.experimental.securecodebox.io/v1" -kind: Scan -metadata: - name: "portscan-berlin-wifi" - label: - office: berlin - vlan: wifi -spec: - cascades: - # Uses all `CascadingRules` in the namespace - matchLabels: {} - name: "nmap" - parameters: ["-sV", "10.42.0.0/16"] -``` - -The label selectors also allow the more powerful [matchExpression](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#set-based-requirement) selectors: - -```yaml -apiVersion: "execution.experimental.securecodebox.io/v1" -kind: Scan +The _secureCodeBox operator_ would process all these CRDs in the namespace of the scan and execute the *read and write* ones first in serial only one at a time to avoid write conflicts and then the *read only* ones in parallel. + +[source,yaml] +---- +apiVersion: execution.experimental.securecodebox.io/v1 +kind: ScanCompletionHook metadata: - name: "example.com" + name: my-metadata spec: - scanType: nmap - parameters: - - -p22,80,443 - - example.com - cascades: - # Using matchExpression instead of matchLabels - matchExpression: - key: "securecodebox.io/intensive" - operator: In - # This select both light and medium intensity rules - values: [light, medium] -``` + type: ReadAndWrite + # If implemented like the current persistence provider + image: my-metadata:v2.0.0 +---- + +The Execution Flow would then look something like this: + +.... + ┌ ReadOnly─Hooks─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ + ┌ ReadAndWriteHooks ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┌────────────────────────────────┐ │ + ┌───────────────────────┐ │ ┌──┼▶│ Elastic PersistenceProvider │ +┌──────────────────┐ ┌──────────────────┐ │ │ ReadAndWrite Hook #1 │ ┌───────────────────────┐ │ └────────────────────────────────┘ │ +│ Scan ├──▶│ Parsing │────▶│ "MyMetaDataProvider" ├─▶│ ReadAndWrite Hook #2 │─┼──┤ │ ┌────────────────────────────────┐ +└──────────────────┘ └──────────────────┘ │ └───────────────────────┘ └───────────────────────┘ └───▶│ DefectDojo PersistenceProvider │ │ + ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ │ └────────────────────────────────┘ + ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ +.... + +====== Pros + +* Only one implementation. +* Pretty generic to expand and test out new ideas without having to modify the _secureCodeBox operator_. + +====== Cons + +* Possibly an "`over-abstraction`". +* Need to refactor the _persistence-elastic_ provider. +* The "`general implementation`" will be harder than the individual ones. + +===== Solution Approach 2: Keep Split between Persistence Provider and MetaData Provider + +Keep _PersistenceProvider_ as they are and introduce new _MetaDataProvider_ CRD which gets executed before the _PersistenceProviders_ by the __secureCodeBox operator_. + +.... + ┌ Persistence Provider─ ─ ─ ─ ─ ─ ─ ─ + ┌ MetaData Provider ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┌────────────────────────────────┐ │ + ┌───────────────────────┐ │ ┌──┼▶│ Elastic PersistenceProvider │ +┌──────────────────┐ ┌──────────────────┐ │ │ ReadAndWrite Hook #1 │ ┌───────────────────────┐ │ └────────────────────────────────┘ │ +│ Scan ├──▶│ Parsing │────▶│ "MyMetaDataProvider" ├─▶│ ReadAndWrite Hook #2 │─┼──┤ │ ┌────────────────────────────────┐ +└──────────────────┘ └──────────────────┘ │ └───────────────────────┘ └───────────────────────┘ └───▶│ DefectDojo PersistenceProvider │ │ + ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ │ └────────────────────────────────┘ + ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ +.... + +====== Pros + +* Quicker to implement. +* Might be worth it to have a separate concept for it. + +====== Cons + +* Not sure if it worth to introduce a new CRD for everything, especially when it's conceptually pretty close to to something already existing. + +==== Question 2: How Should the Execution Model Look like for Each Concept? + +===== Solution Approach 1: Like the Persistence Provider + +Basically a docker container which process findings takes two arguments: + +. A pre-defined URL to download the findings from. +. A pre-defined URL to upload the modified findings to. + +Examples: + +* NodeJS: `node my-metadata.js "https://storage.googleapi.com/..." "https://storage.googleapi.com/..."` +* Java: `java my-metadata.jar "https://storage.googleapi.com/..." "https://storage.googleapi.com/..."` +* Golang: `./my-metadata "https://storage.googleapi.com/..." "https://storage.googleapi.com/..."` + +====== Pros + +* One liner with the current implementations. +* Code overhead / wrapper code is pretty minimal. +* Zero scale: no resource costs when nothing is running. + +===== Cons + +* May results in too many Kubernetes jobs. +** Resource blocking on finished resources. +** `ttlAfterFinished` enabled. +* Container runtime overhead (especially time). + +===== Solution Approach 2: A WebHooks Like Concept + +Analog to kubernetes webhooks: HTTP server receiving findings and returning results. + +===== Pros + +* Milliseconds instead of seconds for processing. +* No overhead for container Creation. +* No additional kubernetes jobs needed. + +===== Cons + +* Introduces new running services which needs to be maintained and have uptime. +* Code overhead / boilerplate (Can be mitigated by an SDK). +* Debugging of individual _MetaDataProvider_ is harder than a single service which handles everything. +* Introduces "`new`"cConcept. +* Certificate management for webhook services (`cert-manager` required by default?). +* Scaling for systems with lots of load could be a problem. +* One service per namespace (multiple tenants) needed -> results in many running active services which is resource consuming. + +== Decision + +Regarding question 1 it seems that both solution approaches are resulting in the same execution model. We decided to implement solution approach 1 and unify both concepts into a more general concept with the name _hook concept_. Therefore we exchange the existing name _PersistenceProvider_ for phase 3 in the execution model with a more general term _processing_: + +.... +┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐ +│ scanning ├─────────▶│ parsing ├─────────▶│ processing │ +│ (Phase 1) │ │ (Phase 2) │ │ (Phase 3) │ +└──────────────────┘ └──────────────────┘ └──────────────────┘ +.... + +Regarding question 2 we decided to implement the solution approach 1 with a job-based approach (no active service component needed). Therefore the phase 3 _processing_ will be split into two separate phases named _ReadAndWriteHooks_ (3.1) and _ReadOnlyHooks_ (3.2) +// #30 to what refers 3.1 and 3.2? + +.... + ┌ 3.2 processing: ReadOnlyHooks ─ ─ ─ + ┌ 3.1 processing: ReadAndWriteHooks ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┌────────────────────────────────┐ │ + ┌───────────────────────┐ │ ┌──┼▶│ Elastic PersistenceProvider │ +┌──────────────────┐ ┌──────────────────┐ │ │ ReadAndWrite Hook #1 │ ┌───────────────────────┐ │ └────────────────────────────────┘ │ +│ scanning ├──▶│ parsing │────▶│ "MyMetaDataProvider" ├─▶│ ReadAndWrite Hook #2 │─┼──┤ │ ┌────────────────────────────────┐ +└──────────────────┘ └──────────────────┘ │ └───────────────────────┘ └───────────────────────┘ └───▶│ DefectDojo PersistenceProvider │ │ + ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ │ └────────────────────────────────┘ + ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ +.... + +== Consequences + +With the new _hook concept_ we open the _phase 3 processing_ to a more intuitive and flexible architecture. It is easier to understand because _WebHooks_ are already a well known concept. It is possible to keep the existing implementation of the _PersistenceProvider_ and integrate them with a lot of other possible processing components in a more general fashion. In the end, this step will result in a lot of additional feature possibilities, which go far beyond the existing ones proposed here. Therefore we only need to implement this concept once in the _secureCodeBox operator_ and new ideas for extending the _DataProcessing_ will not enforce conceptual or architectural changes. + +Ideas for additional processing hooks: + +* Notifier hooks (_ReadOnlyHook_) e.g., for chat (slack, teams etc.), metric, alerting systems +* MetaData enrichment hooks (_ReadAndWriteHook_) +* FilterData hooks (_ReadAndWriteHook_) (e.g., false/positive handling) +* SystemIntegration hooks (_ReadOnlyHook_) e.g., for ticketing systems like Jira +* CascadingScans hooks (_ReadOnlyHook_) e.g., for starting new security scans based on findings diff --git a/docs/adr/adr_0003.adoc b/docs/adr/adr_0003.adoc new file mode 100644 index 00000000..6fb41175 --- /dev/null +++ b/docs/adr/adr_0003.adoc @@ -0,0 +1,182 @@ +[[ADR-0003]] += ADR-0002: How can we introduce a mechanism to start specialized scans on the results of previous scans? + +[cols="h,d",grid=rows,frame=none,stripes=none,caption="Status",%autowidth] +|==== + +| Status +| ACCEPTED + +| Date +| 2020-05-20 + +| Author(s) +| Jannik Hollenbach , + Robert Seedorff , + Sven Strittmatter +|==== + +== Context + +=== Status Quo + +Currently scans by the secureCodeBox are single focused on a specific tool. +Combining multiple scans requires manual or scripting by the user to use the results of a scan (e.g. Nmap) as a input for another scanner (e.g. SSLyze) + +=== Problem and Question + +How can the results of a scan be used to automatically configure subsequent specialized scans for identified targets. + +In general we want to describe cascading scans like: + +``` ++--------+ +--------+ +--------+ +| scan 1 |-- result -->| scan 2 |-- result -->| scan 3 | ++--------+ +--------+ | +--------+ + | + | +--------+ + +---->| scan 4 | + +--------+ +```` + +A concrete example: + +``` ++----------------+ +-----------------+ +-----------+ +| <> | | <> | | <> | +| find all hosts |-- IP -->| find open ports |-- port 443 -->| check TLS | ++----------------+ +-----------------+ | +-----------+ + | + | +-------------+ + | | <> | + +------->| check HTTPd | + +-------------+ +``` + +The solution should fulfill the following criteria: + +- The "rules" used to describe which subsequent scans can be executed should be modular, so that they can be packaged together with the scan types. +- It should be possible for a user to select which scan rules should be applied +- Protections should be in place to ensure that the clusters are not completely overwhelmed by these automatically created scans. Especially circular structures which create a infinite number of scans should be prevented. + +== Decision + +It was decided to implement these rules as Custom Resource Definitions (CRDs) in Kubernetes. +This allows the Helm Charts of the scanners to package related rules for the scanner together with their ScanTypes. + +=== Defining CascadingRule + +The so called "CascadingRules" consist of a "matches" section which contains one or multiple rules which are compared against findings. +When a finding matches a rule the "scanSpec" section will then be used to create a new scan. +To customize the scan to match the finding, the [mustache](https://github.com/janl/mustache.js) templating language can be used to reference fields of the finding. + +```yaml +apiVersion: "cascading.experimental.securecodebox.io/v1" +kind: CascadingRule +metadata: + name: "tls-scans" + labels: + # Described how "invasive" the scan is. + # Possible values: "invasive" or "non-invasive" + # CascadingRules are considered "invasive" when the Scan they start actively sends out packages with attack payloads. + securecodebox.io/invasive: non-invasive + # Described the intensiveness level on a scanning and computational resource level. + # Possible values: "ligh", "medium", "intense" + # CascadingRules are considered more "intensive" when the Scan they start consumes lots of computational resources like RAM, CPU, or Network + securecodebox.io/intensive: light +spec: + matches: + # CascadingRule triggers if a finding matches at least one of the anyOf matchers + # With the first version of this implementation only anyOf would be supported. + # If this turns out to be lacking and other operators (like `allOf` can be introduced without breaking changes) + anyOf: + # define an explicit "port" as finding and a given port number + - category: "Open Port" + attributes: + port: 443 + service: "https" + # define an "port service" finding (any port) + - category: "Open Port" + attributes: + service: "https" + scanSpec: + name: "sslyze" + parameters: ["--regular", "{{attributes.hostname}}"] +``` + +=== Using CascadingRules + +By default no cascading Rules will be used. + +```yaml +# Nmap Scan without cascading rules +apiVersion: "execution.experimental.securecodebox.io/v1" +kind: Scan +metadata: + name: "portscan-berlin-wifi" + label: + office: berlin + vlan: wifi +spec: + name: "nmap" + parameters: ["-sV", "10.42.0.0/16"] +``` + +To enable cascading rules you need to specify a label selector to select the cascading rules you'd like + +```yaml +apiVersion: "execution.experimental.securecodebox.io/v1" +kind: Scan +metadata: + name: "portscan-berlin-wifi" + label: + office: berlin + vlan: wifi +spec: + cascades: + matchLabels: + # Uses all CascadingRules in the namespace which are labelled as "non-invasive" and a intensiveness level of "light" + securecodebox.io/invasive: non-invasive + securecodebox.io/intensive: light + name: "nmap" + parameters: ["-sV", "10.42.0.0/16"] +``` + +To implicitly enable all cascading rules (not-recommended) a empty label selector can be used + +```yaml +apiVersion: "execution.experimental.securecodebox.io/v1" +kind: Scan +metadata: + name: "portscan-berlin-wifi" + label: + office: berlin + vlan: wifi +spec: + cascades: + # Uses all `CascadingRules` in the namespace + matchLabels: {} + name: "nmap" + parameters: ["-sV", "10.42.0.0/16"] +``` + +The label selectors also allow the more powerful [matchExpression](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#set-based-requirement) selectors: + +```yaml +apiVersion: "execution.experimental.securecodebox.io/v1" +kind: Scan +metadata: + name: "example.com" +spec: + scanType: nmap + parameters: + - -p22,80,443 + - example.com + cascades: + # Using matchExpression instead of matchLabels + matchExpression: + key: "securecodebox.io/intensive" + operator: In + # This select both light and medium intensity rules + values: [light, medium] +``` From d0994d73edbf8ab143c6de1e3e0de76f47ebdbb0 Mon Sep 17 00:00:00 2001 From: Daniel Patanin Date: Tue, 29 Sep 2020 10:15:46 +0200 Subject: [PATCH 2/4] Add adr file about new documentation framework --- docs/adr/adr_0002.adoc | 2 +- docs/adr/adr_0003.adoc | 2 +- docs/adr/adr_0004.adoc | 41 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 43 insertions(+), 2 deletions(-) create mode 100644 docs/adr/adr_0004.adoc diff --git a/docs/adr/adr_0002.adoc b/docs/adr/adr_0002.adoc index 60f6eba1..8476fffb 100644 --- a/docs/adr/adr_0002.adoc +++ b/docs/adr/adr_0002.adoc @@ -1,5 +1,5 @@ [[ADR-0002]] -= ADR-0000: How can we introduce a more general extension concept for data processing modules? += ADR-0002: How can we introduce a more general extension concept for data processing modules? [cols="h,d",grid=rows,frame=none,stripes=none,caption="Status",%autowidth] |==== diff --git a/docs/adr/adr_0003.adoc b/docs/adr/adr_0003.adoc index 6fb41175..3d7d0000 100644 --- a/docs/adr/adr_0003.adoc +++ b/docs/adr/adr_0003.adoc @@ -1,5 +1,5 @@ [[ADR-0003]] -= ADR-0002: How can we introduce a mechanism to start specialized scans on the results of previous scans? += ADR-0003: How can we introduce a mechanism to start specialized scans on the results of previous scans? [cols="h,d",grid=rows,frame=none,stripes=none,caption="Status",%autowidth] |==== diff --git a/docs/adr/adr_0004.adoc b/docs/adr/adr_0004.adoc new file mode 100644 index 00000000..08bc5c83 --- /dev/null +++ b/docs/adr/adr_0004.adoc @@ -0,0 +1,41 @@ +[[ADR-0004]] += ADR-0004: Which framework could be more useful for documentation purposes? + +[cols="h,d",grid=rows,frame=none,stripes=none,caption="Status",%autowidth] +|==== + +| Status +| ACCEPTED + +| Date +| 2020-09-29 + +| Author(s) +| Daniel Patanin daniel.patanin@iteratec.com + +|==== + +== Context + +Since the last major update of our GatsbyJS based website, the original requirements for our ideal webpage have changed. +Originally we needed a website, which could present our project briefly and hold the documentation, dynamically build from our main repository. +The first aspect has changed quite severe since we aim for an actual marketing oriented homepage now. The second goal stays the same, but changed contextually. +We no longer just want to host the documentation, but furthermore want to stay on modern documentation standards and since our team does not want to invest too +much time into the documentation site's development and maintenance, we are looking for a component library or whole framework specialized on documentation. + +== Decision + +We did not evaluate many different frameworks. The journey was quite short in fact. The first documentation oriented framework we looked at was Docusaurus. +As I progressed into trying out to build a basic Docusaurus site and copied all documentation files into it, the website actually looked nice and quite finished already, +after only a few hours of tinkering. It is that easy and simple to understand, especially with it's own good documentation. After reviewing this very basic website, +we decided to advance this and already try to retrieve the documentation remotely from our main repository. At this point the decision was pretty clear already: If there +is no major drawback coming up, Docusaurus is our new framework of choice. And there was no drawback major enough to revoke this decision (yet). Everything we programmatically +wanted to achieve, we could do so by basic scripting and we never really have to think about components like the sidebar, navbar etc. since this is all done completely automatically +by Docusaurus itself. + +== Consequences + +As simple and easy it sounds, so it is, meaning that since the build is automated to a very big degree, we are 'forced' to follow Docusaurus' exact guidelines and mechanics. +For now they are almost identical to what we require. And of course we can build custom components and custom pages, but the documentation and blog part of the site are very strict. +Since we want to use this website mainly (ideally only) for documentation purposes, we need a separate marketing page (more on that in <>). As far as maintenance goes, +we mostly need to maintain our custom build scripts. Everything else is very much automated. From f73bf64da448df8eafba98b0fae04dc0fbeb4d83 Mon Sep 17 00:00:00 2001 From: Daniel Patanin Date: Tue, 29 Sep 2020 11:32:18 +0200 Subject: [PATCH 3/4] Add adr for proposing an one-pager This adr sums up roughly what was proposed and should be looked over very good. --- docs/adr/adr_0005.adoc | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100644 docs/adr/adr_0005.adoc diff --git a/docs/adr/adr_0005.adoc b/docs/adr/adr_0005.adoc new file mode 100644 index 00000000..94ca4a2b --- /dev/null +++ b/docs/adr/adr_0005.adoc @@ -0,0 +1,41 @@ +[[ADR-0000]] += ADR-0005: Buying separate marketing oriented webpage + +[cols="h,d",grid=rows,frame=none,stripes=none,caption="Status",%autowidth] +|==== +| Status +| PROPOSED + +| Date +| 2020-09-29 + +| Author(s) +| Daniel Patanin daniel.patanin@iteratec.com +|==== + +== Context + +As our requirements about our homepage changed we no longer just want to give a simple overview about our project on our landing page as we did with our GatsbyJS based website, +but furthermore we require a marketing oriented webpage additionally to our documentation site (for more on that see <>). This marketing page should satisfy following aspects: + +- Modern, ideally timeless, design +- Overview about the secureCodeBox +- Display of our project's advantages +- Offer for service (prizing) +- Encouragement for contacting us +- Encouragement for contributing + +== Decision + +Since there is no arguable reason to build such a one-pager ourselves, we should buy a ready to go solution for us. We are not aiming to invest extensive time neither in building such a page, +nor in gaining the desired design and marketing expertise, and also not hiring experts on this field (all this would actually be more expensive and time consuming than buying a site). +Some stores for such pages are: + +- https://www.templatemonster.com/ +- https://onepagelove.com/templates +- https://themeforest.net/tags/onepage + +== Consequences + +If we want to change the style or design someday, we probably could not reuse the bought template if it is not a minor change, thus would need to buy another template. +We also would have to orient the presentation of the secureCodeBox around the template we decide on, but nowadays we have a ginormous variety to choose from. From 1943ffe1d6641fe029fc892fac7038491cc2149d Mon Sep 17 00:00:00 2001 From: Daniel Patanin <44839597+dpatanin@users.noreply.github.com> Date: Tue, 29 Sep 2020 16:12:51 +0200 Subject: [PATCH 4/4] Update references - Add reference to superseded adr - Remove chronologically incorrect reference --- docs/adr/adr_0004.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/adr/adr_0004.adoc b/docs/adr/adr_0004.adoc index 08bc5c83..940a6b53 100644 --- a/docs/adr/adr_0004.adoc +++ b/docs/adr/adr_0004.adoc @@ -5,7 +5,7 @@ |==== | Status -| ACCEPTED +| ACCEPTED | SUPERSEDES <> | Date | 2020-09-29 @@ -37,5 +37,5 @@ by Docusaurus itself. As simple and easy it sounds, so it is, meaning that since the build is automated to a very big degree, we are 'forced' to follow Docusaurus' exact guidelines and mechanics. For now they are almost identical to what we require. And of course we can build custom components and custom pages, but the documentation and blog part of the site are very strict. -Since we want to use this website mainly (ideally only) for documentation purposes, we need a separate marketing page (more on that in <>). As far as maintenance goes, +Since we want to use this website mainly (ideally only) for documentation purposes, we need a separate marketing page. As far as maintenance goes, we mostly need to maintain our custom build scripts. Everything else is very much automated.