This repository has been archived by the owner on Dec 1, 2023. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
12 changed files
with
160 additions
and
92 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
aetherfs.tech |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,46 @@ | ||
AetherFS assists in the production, distribution, and replication of embedded databases and in-memory datasets. It | ||
provides engineers with a platform to manage collections files called datasets. AetherFS optimizes its use of the | ||
underlying blob store (AWS S3 or equivalent) to reduce cost to operators and improve performance for end users. | ||
|
||
_Why not use S3 directly or a file server?_ | ||
|
||
While this is an option, there are several problems that arise with this solution. For example, to produce two | ||
references to the same dataset, you must upload the same set of files twice. If you want to produce three references, | ||
then three times (and so on). This comes at a cost of additional time in your pipeline and storage costs. | ||
|
||
Instead, producers tag datasets in AetherFS. A tag can refer to a specific version ([semantic][] or [calendar][]) or a | ||
channel that consumers can subscribe to (`latest`, `stable`, etc.). Instead of storing entire snapshots of datasets | ||
in each version, AetherFS removes duplicated blocks between them. This allows clients to re-use blocks of data and only | ||
download new or updated portions. | ||
|
||
[semantic]: https://semver.org | ||
[calendar]: https://calver.org | ||
|
||
## Status | ||
|
||
This project is under active development. | ||
|
||
- Documentation | ||
- Architecture Document | ||
- Features | ||
- HTTP file server for ease of interaction | ||
- REST and gRPC APIs for programmatic interaction | ||
- Optional agent that can manage a shared file systems | ||
- Efficiently persist and query information stored in [AWS S3][] (or compatible) | ||
- Authenticate using common schemes (such as OIDC) | ||
- Enforce access control around datasets | ||
- Encrypt data in transit and at rest | ||
- Built-in developer tools to help understand dataset performance and usage | ||
|
||
[AWS S3]: https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html | ||
|
||
|
||
### Expectations | ||
|
||
This is a project I'm mostly iterating on in my free time. It's closed source, and I have no intent to open source. If | ||
you're interested in learning more or getting updates, please sign up using the link below. | ||
|
||
|
||
[![Project Interest Form][]](https://forms.gle/uCMy38ZLEchfNuka9) | ||
|
||
[Project Interest Form]: https://img.shields.io/badge/-Project%20Interest%20Form-blue?style=for-the-badge |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
theme: jekyll-theme-minimal | ||
exclude: | ||
- AUTHORS | ||
- LICENSE | ||
title: AetherFS | ||
description: A virtual file system for small to medium sized datasets (MB or GB, not TB or PB). |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,8 +1,15 @@ | ||
package blocks | ||
|
||
var ( | ||
KiB int32 = 1 << 10 | ||
MiB int32 = 1 << 20 | ||
type Size int32 | ||
|
||
const ( | ||
Byte Size = 1 << (10 * iota) | ||
Kibibyte | ||
Mebibyte | ||
) | ||
|
||
PartSize = 64 * KiB | ||
var ( | ||
// PartSize is a cache-optimized length that is used to send and share parts of a block amongst a group of nodes. | ||
// It is also used during uploads and downloads as the segment sizes to avoid buffering gigabytes of data in memory. | ||
PartSize = 64 * Kibibyte | ||
) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
package components | ||
|
||
import ( | ||
"github.com/gin-gonic/gin" | ||
"google.golang.org/grpc/metadata" | ||
) | ||
|
||
// TranslateHeadersToMetadata provides a gin handler that will copy Headers from an HTTP request to gRPC headers. | ||
func TranslateHeadersToMetadata() gin.HandlerFunc { | ||
return func(ginctx *gin.Context) { | ||
// preprocess headers into grpc metadata | ||
md := metadata.New(nil) | ||
for k, vv := range ginctx.Request.Header { | ||
md.Set(k, vv...) | ||
} | ||
|
||
ctx := metadata.NewIncomingContext(ginctx.Request.Context(), md) | ||
ginctx.Request = ginctx.Request.WithContext(ctx) | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters