Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor KES API and internals #403

Merged
merged 5 commits into from
Oct 24, 2023
Merged

refactor KES API and internals #403

merged 5 commits into from
Oct 24, 2023

Conversation

aead
Copy link
Member

@aead aead commented Oct 19, 2023

Description:

This commit introduces a series of significant changes to various components within the KES project. Among other things it:

  1. Exposes a top-level library API for running and customizing KES servers.
  2. Improves logging by using structured logging (log/slog).
  3. Removes unused code
  4. Introduces a KES-specific framework for handling HTTP requests (internal/api).
  5. Stabilizes the KES API and prepares the introduction of protobuf as serialization format (in addition to JSON).

However, this commit does not refactor the kv package or the KES config file handling. While still required, this will be done in a separate commit.

Performance

A lot of effort has gone into designing and implementing an efficient KES library API. Since majority of KES operations are read-only, accessing a policy, encrypting a message, a.s.o., it can leverage and benefit from lock-free concurrency primitives. Hence, the Server type tries to avoid blocking on sync.{RW}Mutex as much as possible and instead uses atomic primitives, like atomic.Pointer.

Further, the logging framework has been completely reworked to use structured logging using the log/slog standard library package. Now, error log messages are only generated when required (based on log levels). The audit logging framework (AuditHandler and AuditRecord type) works similar to the slog package and is also designed to be efficient.

Readability

The new internal/api package provides a small KES-specific framework for defining HTTP APIs and handling request. It tries to provide composable primitives to build HTTP APIs that are efficient, secure and easy to reason about. It provides a specific Request type that represents an authenticated HTTP request. This allows to separate buisness logic (e.g. handling a key creation request) from timeout handling, authentication, etc.

Further, this commit tries to add more expressive documentation describing the intent.

Versioning

The KES library package will follow semantic versioning, like any other Go module. However, the KES server command and CLI (cmd/kes) will continue to use the rolling release timestamp versioning. A KES library release can be tagged independently from the KES CLI and vice versa. Users of the KES package will be able to import like any other Go module: import "github.com/minio/kes@v0.24.0".

**Description:**

This commit introduces a series of significant changes to various
components within the KES project. Among other things it:
1. Exposes a top-level library API for running and customizing
   KES servers.
2. Improves logging by using structured logging (`log/slog`).
3. Removes unused code
4. Introduces a KES-specific framework for handling HTTP requests
   (`internal/api`).
5. Stabilizes the KES API and prepares the introduction of
   protobuf as serialization format (in addition to JSON).

However, this commit does not refactor the `kv` package or
the KES config file handling. While still required, this will
be done in a separate commit.

**Performance**

A lot of effort has gone into designing and implementing an efficient
KES library API. Since majority of KES operations are read-only,
accessing a policy, encrypting a message, a.s.o., it can leverage and
benefit from lock-free concurrency primitives. Hence, the `Server`
type tries to avoid blocking on `sync.{RW}Mutex` as much as possible
and instead uses atomic primitives, like `atomic.Pointer`.

Further, the logging framework has been completely reworked to use
structured logging using the `log/slog` standard library package.
Now, error log messages are only generated when required (based on
log levels). The audit logging framework (`AuditHandler` and
`AuditRecord` type) works similar to the `slog` package and is also
designed to be efficient.

**Readability**

The new `internal/api` package provides a small KES-specific framework
for defining HTTP APIs and handling request. It tries to provide
composable primitives to build HTTP APIs that are efficient, secure
and easy to reason about. It provides a specific `Request` type that
represents an authenticated HTTP request. This allows to separate
buisness logic (e.g. handling a key creation request) from timeout
handling, authentication, etc.

Further, this commit tries to add more expressive documentation
describing the intent.

**Versioning**

The KES library package will follow semantic versioning, like any
other Go module. However, the KES server command and CLI (`cmd/kes`)
will continue to use the rolling release timestamp versioning.
A KES library release can be tagged independently from the KES
CLI and vice versa. Users of the KES package will be able to
import like any other Go module: `import "github.com/minio/kes@v0.24.0"`.

Signed-off-by: Andreas Auernhammer <github@aead.dev>
We need to check that the added `io.Writer`
is not nil.

Signed-off-by: Andreas Auernhammer <github@aead.dev>
variable s is initialize on all execution paths, so we might as well just initialize it at the start of the function

Signed-off-by: Sveinn <info@sveinn.is>
zveinn
zveinn previously approved these changes Oct 24, 2023
Copy link
Contributor

@zveinn zveinn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

made a tiny change but it's mostly cosmetic, everything looks good.

shtripat
shtripat previously approved these changes Oct 24, 2023
Copy link
Contributor

@shtripat shtripat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have some minor comments for spell checks

audit.go Outdated Show resolved Hide resolved
config.go Show resolved Hide resolved
config.go Outdated Show resolved Hide resolved
internal/api/api.go Outdated Show resolved Hide resolved
internal/api/api.go Outdated Show resolved Hide resolved
internal/api/request.go Show resolved Hide resolved
keystore.go Outdated Show resolved Hide resolved
keystore.go Show resolved Hide resolved
keystore.go Outdated Show resolved Hide resolved
server.go Outdated Show resolved Hide resolved
@aead aead dismissed stale reviews from shtripat and zveinn via 93e31f8 October 24, 2023 15:59
Signed-off-by: Andreas Auernhammer <github@aead.dev>
@aead aead merged commit 0e49f95 into master Oct 24, 2023
9 checks passed
@aead aead deleted the api-refactor branch October 24, 2023 18:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants