-
-
Notifications
You must be signed in to change notification settings - Fork 35
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Issue 370 (and 423) #426
Merged
Merged
Issue 370 (and 423) #426
Changes from 121 commits
Commits
Show all changes
125 commits
Select commit
Hold shift + click to select a range
772cc19
Read pre-0.9.0 data and populate CaObjectsStore (#370)
2d350c9
Build up CaObjects following changes in CAs (in paralel so-far). #370
f16c798
Use a HashMap with json mapping. (#370)
bf35238
Re-issue manifests and CRLs in CaObjectStore. (#370)
a3fc486
Sync elements from CaObjectsStore to repository. (#370)
fb70e17
Schedule repo sync on mft/crl reissue (#370)
45038cd
Migrate pre 0.9.0-rc1 commands and events. Remove old publication cod…
74b9bf7
Remove ManifestInfo (no longer used) (#370)
3ec76b8
Remove CrlInfo (no longer used) (#370)
faa814d
Remove types no longer used in publication code (#370)
bc9b24e
Don't duplicate data - keyidentifier is in the stored certificate.
d18d0e1
Improve json representation of stored commands. (#370)
3a72f25
Improve json representation of stored events. (#370)
0e551ca
Re-issue ROAs about to expire (no longer side-effect of MFT/CRL publi…
c5bb96d
Remove archiving code for commands which will now be removed (we will…
6c212ec
Better json for maps (using explicit key) and add deterministic order…
39da2a1
Migrate repository by doing a keyroll. (#370)
a033a77
Merge branch 'v0.8.1-bis' into issue-370
d395352
Increase time for testing that objects get published.
e9db021
Fix parsing of existing CA handles for CaObjectStore.
75c9b05
Use a swap file when writing (avoid corrupt json if disk is full) (#370)
bc582bb
Typo correction in comment: representaion -> representation
ximon18 d3fdbbd
Pubd: use state for repo content, rather than events. #423
2a39a12
Initial commit of Krill development markdown documentation.
4f05d04
Describe code for starting / running Krill.
000b74a
Use chapters for dev doc. Clarify daemon code layers.
10a9697
When upgrading check version *before* and avoid always upgrading to t…
c456447
Add documentation on CLI code.
188deb1
Update documentation caption style.
2db0101
Add some background documentation on event sourcing concepts.
6b82c46
Remove unused concepts from the eventsourcing code.
d47f312
Don't insert a line break before publisher metrics if no publisher me…
ximon18 0f08d93
Fix json for storedeffect to be explicit.
f8989f2
Improve naming and comments for eventsourcing types.
8ad0160
Add documentation of Krill event sourcing.
3097854
Add documentation on KeyValueStore.
768c2de
Improve code readability and rename PubServer to RepositoryManager (#…
547c2e3
Fix style and clarify comments.
94cf61e
Clarify responsibility of RepositoryManager.
24586f0
Cleanup.
bfc2f8a
Add documentation on RepositoryManager.
922c778
Clarify documentation.
8c0acbd
Rename types to match 'RepositoryManager'
e700a9f
Enable testbed test by default - and fix the issue it found!
9eb970d
Do not upgrade empty keystores.
389bbda
Include published objects in repository stats (again).
9dea030
Typo correction: ir -> is
ximon18 2163f17
Typo correction: is -> if
ximon18 029f83b
typos
density215 4aee4a5
Missing word "to".
ximon18 7769f15
Correction: "to the it" -> "to it"
ximon18 fc2726a
Missing "the".
ximon18 5cc832c
Enable Rust syntax higlighting for storable trait snippet.
ximon18 c0ab0eb
Typo correction: "an" -> "and"
ximon18 90a6366
Update Lagosta for changed repository status JSON.
506c100
Typo correction: RF -> RFC
ximon18 af348a9
typo + uppercasing
density215 bbf7f02
Return published objects in repo status when embedded repo is used.
6e91e09
typos
density215 7283f1b
Clarify versioning, locking and transactions.
0ccd06c
Include relative link to source path.
eef5570
Include *correct* link to source path.
711bae9
Exclude duplicate authorization updates submitted through the API.
61cd505
Update openapi spec for changed "ParentCaContact" json.
2ac06e4
Fix comment.
17da262
Typo
fcf9f2e
Treat publication failure as internal server error.
d75493b
Rename storage dirs to be consistent and less confusing.
75e4e68
Change pending_key to pending_key_id for clarity.
cad093d
Minor fixes based on PR review.
f5bc31a
Include 'sorted' in serde module names.
c298c2f
Fix error messages for failing test.
ee157f5
Merge branch 'v0.8.1-bis' into issue-370. Includes Lagosta master com…
ximon18 0bcd1be
Rename CaServer to CaManager
597a6f3
Increase time to check for publication in test to 10 minutes - becaus…
31eec03
Wait for all keys to become a certain state in testing.
dc100dd
Merge branch 'v0.8.1-bis' into issue-370.
ximon18 361d94b
Typo: KRILL_ENV_TESTBED_RSYNC -> KRILL_ENV_TESTBED_RRDP
ximon18 cf6263d
Force republish in functional.rs - it looks like a message queue timi…
62e0f74
Add CaManager doc - and some resulting code cleanup.
c7e6924
Comment and document CaManager.
9a3797b
Document and clean code for CaManager.
5a09749
Krill creates a notification.xml, not a notify.xml.
ximon18 110b195
Typo correction: typed -> type
ximon18 d9e645c
Merge branch 'v0.8.1-bis' into issue-370
ximon18 099893f
Document CaManager CAs as children functions.
798b46d
Clean up and document CA parent sync.
b847209
FIX: quick hack to match the API change on the server side. A proper …
ximon18 e0f5fd6
Reorder functions according to category.
96e3204
Clean up and document CA as parent code.
87b9fa6
Document CA ROA functions.
2211387
Make 'embedded' mode a first class API citizen for more abstract clie…
ximon18 95edc97
Experimental change to see if it helps with the new 'mft not found in…
ximon18 0743bba
Document key rolls (RFC 6489)
a8d2d9d
Typo - well British ingalish
2840b76
Add CA History pointers in documentation.
01521eb
Rename pubserver module directory.
47849df
Make removing publisher content idempotent for publishers already rem…
5993ec8
Clarify comment.
44755db
Also test the Routinator unstable Docker image (aka Routinator main).
ximon18 e27266b
Attempt to resolve the missing MFT error occasionally reported by Rou…
ximon18 afe81f2
Typo correction: is -> it
ximon18 f6f63c7
Minor English correction
ximon18 2319cda
Minor rewording.
ximon18 f606606
Typo correction: as -> is
ximon18 1189976
Typo correction: as -> a
ximon18 4b6baf3
Minor English change
ximon18 e4bdbd8
Remove errant blank line.
ximon18 15063e5
Singular -> plural
ximon18 ba23509
Linkify issue number.
ximon18 cd3108f
Remove errant blank line.
ximon18 c179df1
Fix RFC reference in docs
9ef1e17
Clarify the use singleton CaManager and RepositoryManager.
ef67f53
Better say cloned, not copied, to avoid further confusion.
f05c392
Grammar!
f92fb2c
Allow resources and child ID to be updated together.
ed6090c
Typo correction: issues -> issued
ximon18 9bad4f9
Missing period character.
ximon18 8047c74
Typo correction: Certifcate -> Certificate
ximon18 cd1f325
Typo: "there there" -> "there".
ximon18 6867621
Process all events post save as a single transaction.
ea8abc1
Fix comments.
bffb078
Clean up code based on review.
df72995
Improve user facing messages.
163ad0d
Fix comment.
File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
The table of contents is too big for display.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,93 @@ | ||
Krill Daemon Setup | ||
================== | ||
|
||
Overview | ||
-------- | ||
|
||
Here we will explain how the Krill daemon is layered and handles requests. This layering works the | ||
same way whether the daemon is running as a Certification Authority, Publication Server, or both. The | ||
components described here are responsible for: | ||
* Parsing configuration | ||
* Starting Krill | ||
* Triggering and executing data migrations on upgrade | ||
* Handling HTTPS requests | ||
* Handling authorization | ||
* Background jobs | ||
|
||
Ultimately the actual requests coming from either the API or background jobs are dispatched to either | ||
the `CaServer` or `RepositoryManager` which are set up using the provided config (e.g. instructing these components | ||
where their data is stored). Theoretically those components could also be wrapped in a different way in | ||
the future, e.g. to support serverless setups using AWS Lambda functions, provided of course that authorization, | ||
configuration, and concurrency are handled. | ||
|
||
Binaries | ||
-------- | ||
|
||
The project includes two binaries which can be used to start a Krill daemon. These binaries are fairly | ||
thin executables which are responsible for parsing a configuration file, setting the operation mode, and | ||
then starting the `HTTPS Server` which includes the real `KrillServer`. | ||
|
||
Typically the `krill` binary is used to start Krill as a Certification Authority server, while `krillpubd` | ||
is used to start it as a dedicated Publication Server. That said, mixed operation is also possible as we | ||
will explain below. | ||
|
||
|
||
HTTPS Server | ||
------------ | ||
|
||
Krill uses [hyper](https://hyper.rs/) as an HTTPS server. The set up for this is done in the `start_krill_daemon` | ||
function in `src/daemon/http/server.rs`. This function performs the following steps: | ||
|
||
* Creates the PID file. | ||
* Verifies that the configured data directory is usable. | ||
* Calls 'pre-start' upgrades before state is built. (e.g. migrate data structures). | ||
* Instantiates a `KrillServer`, which will guard all state. | ||
* Creates a self-signed TLS certificate, unless one was prepared earlier. | ||
* Builds a `hyper` server which then connects to the configured port and handles connections. | ||
* This server keeps running until the Krill binary is terminated. | ||
|
||
Note that the `hyper` server itself is stateless. For this it relies on an `Arc<KrillServer>` which can | ||
be cloned cheaply whenever a request is processed. So, we use hyper for the following: | ||
* Get authentication/authorization information from the request (header/cookies dependent on config) | ||
* Serve static content for the Krill UI. | ||
* Map requests to API code in `KrillServer` and serve responses | ||
|
||
> Note that for higher level testing we bypass the Krill binaries, and call the function to start the | ||
> HTTPS server directly, with appropriate configuration settings. Have a look at `tests/functional.rs` | ||
> for an example. | ||
|
||
|
||
KrillServer | ||
----------- | ||
|
||
This is the main daemon component that runs Krill. It won't do actual processing, but it is responsible for running and | ||
mapping calls to the following components (we will describe each component in more detail later): | ||
|
||
| Element | Code Path | Responsibility | | ||
| ------------------- | ----------------------------- | ----------------------------------------------------------- | | ||
| `CaServer` | src/daemon/ca/server.rs | Manages Krill CAs. | | ||
| `RepositoryManager` | src/pubd/pubserver.rs | Manages access to and content of the repository. | | ||
| `Scheduler` | src/daemon/scheduler.rs | Schedules and executes background jobs. | | ||
| `Authorizer` | src/daemon/auth/authorizer.rs | Verifies authentication and authorization for API requests. | | ||
| `BgpAnalyser` | src/commons/bgp/analyser.rs | Compares authorizations to BGP, downloads RIS whois dumps. | | ||
|
||
|
||
KrillMode | ||
--------- | ||
|
||
The `KrillServer` elements are initialized based on which ```KrillMode``` is selected. The following modes are possible: | ||
|
||
| KrillMode | Operation | | ||
|-|-| | ||
| Pubd | The KrillServer will have Some(PubServer), but no (None) CaServer | | ||
| Ca | The KrillServer will have Some(CaServer), but no (None) PubServer | | ||
| Mixed | The KrillServer will have both a CaServer and a PubServer | | ||
| Testbed | Krill runs in test mode. It will have a PubServer, CaServer **AND** an embedded TA | | ||
|
||
If Krill is started with the `krillpubd` binary, then the mode will always be ```KrillMode::Pubd```. If it is started with the | ||
`krill` binary, then the mode will *normally* be ```KrillMode::Ca```. However, for backward compatibility with existing deployments, | ||
the KrillServer will change this mode to ```KrillMode::Mixed``` if it finds that a data directory exists for an initialized | ||
Publication Server with at least one active `Publisher`. ```KrillMode::Testbed``` can be forced if the user sets the URIs for the test | ||
Publication Server rsync and RRDP URI base, using the following two environment variables: `KRILL_TESTBED_RSYNC` and `KRILL_TESTBED_RRDP`. | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,30 @@ | ||
Krill Command Line Client Setup | ||
=============================== | ||
|
||
There are two CLI binaries included in Krill: `krillc` is intended to manage Certification | ||
Authorities, and `krillpubc` is used to manage a Publication Server. | ||
|
||
Essentially the CLIs are a small convenient way to access the Krill API and represent responses | ||
to the user. They parse command line arguments and/or files supplied by the user (where applicable), | ||
and query or post (JSON) to the appropriate API end-point. Responses can be displayed as JSON, or | ||
plain text. | ||
|
||
From a development point of view it's important to know that the argument parsing by the CLIs | ||
is tested manually. This can lead to issues as there is no strong typing enforced by the clapper | ||
library that we use. So: CHECK whenever there are changes. | ||
|
||
What **is** tested properly is the underlying code used by the CLIs to submit data and process | ||
server responses. Our test code bypasses the command line parsing, but it uses the same underlying | ||
code in the higher level tests such as `tests/functional.rs` in order to interact with a running | ||
Krill instance. | ||
|
||
The code can be found under `src/cli`. An overview of the most important elements follows: | ||
|
||
| Element | Code Path | Responsibility | | ||
|---------------------|------------------------------|----------------------------------------------------------------------| | ||
| `KrillClient` | src/cli/client.rs | The client code for Krill CA operations. | | ||
| `KrillPubdClient` | src/cli/client.rs | The client code for Krill Publication Server operations. | | ||
| `Command` | src/cli/options.rs | Enum for the intended CA command. | | ||
| `PublishersCommand` | src/cli/options.rs | Enum for the intended Publication Server command. | | ||
| `ApiResponse` | src/cli/report.rs | Structure to represent API responses. | | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,60 @@ | ||
Event Sourcing Overview | ||
======================= | ||
|
||
The Audit is the Truth | ||
---------------------- | ||
|
||
Event Sourcing is a technique based on the concept that the current state of a thing is | ||
based on a full record of past events - which can be replayed - rather than saving / loading | ||
the state using something like object-relational-mapping or serialization. | ||
|
||
A commonly used example to explain this concept is bookkeeping. In bookkeeping one tracks | ||
all financial transactions, like money coming in and going out of an account. The current | ||
state of an account is determined by the culmination of all these changes over time. Using | ||
this approach, and assuming that no one altered the historical records, we can be 100% certain | ||
that the current state reflects reality. Conveniently, we also have a full audit trail which | ||
can be shown. | ||
|
||
|
||
CQRS and Domain Driven Design | ||
----------------------------- | ||
|
||
Event Sourcing is often combined with CQRS: Command Query Responsibility Segregation. The | ||
idea here is that there is a separation between intended changes (commands) sent to your | ||
entity, the internal state of the entity, and its external representation. | ||
|
||
Separating these concerns we can also borrow heavily from ["Domain Driven Design"](https://en.wikipedia.org/wiki/Domain-driven_design). | ||
In DDD structures are organized into so-called of "aggregates" of related structures. At | ||
their root they are joined by an "aggregate root". | ||
|
||
Commands, Events and Data | ||
------------------------- | ||
|
||
This separation means that the aggregate is a bit like a black box to the outside world, | ||
but in a positive sense. Users of the code just need to know what messages of intent (commands) | ||
they can send to it. This interface, like an API, should be fairly stable. | ||
|
||
The first thing to do when a command is sent is that an aggregate is retrieved from storage, | ||
so that it can receive a command. The state of the aggregate is the result of all past | ||
events - but, because this can get slow, aggregate snapshots are often used for efficiency. | ||
If a snapshot does not include the latest events, then they are simply re-applied to it. | ||
|
||
When an aggregate receives a command it can see if a change can be applied. It is fully | ||
in charge of its own consistency. If it isn't then the aggregate root is usually at the | ||
wrong level. The result of the command can either be an error - the command is rejected - or | ||
a number of events which represent state changes are applied to the aggregate. | ||
|
||
When events are applied they are also saved, so that they can be replayed later. Furthermore, | ||
implementations often use message queues where events are posted as well. This allows other | ||
components in the code to be triggered by changes in an aggregate. | ||
|
||
In its purest form (hint: we don't do this), these messages can then be used to (re-)generate | ||
one or more data representations that users can *query*. E.g. you could populate data tables | ||
in a SQL database if that floats your boat. | ||
|
||
Credits / Read More.. | ||
--------------------- | ||
|
||
This combination of techniques has been championed by various people, most notably Greg Young | ||
and Martin Fowler. You can do your own internet search find out much more about how this can | ||
work, and how it is done in other projects. |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is not really anything to do with this PR, I just snuck it in here as I saw that the dependency can still automatically pick up the next alpha which could contain breaking changes and so should be pinned to the .3 version which is known to work with our code.