-
Notifications
You must be signed in to change notification settings - Fork 4
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
1 changed file
with
13 additions
and
33 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,47 +1,27 @@ | ||
# GoBox | ||
|
||
## Setup | ||
- Must have `GOBOX_AWS_ACCESS_KEY_ID` and `GOBOX_AWS_SECRET_ACCESS_KEY` set for aws client. | ||
GoBox is a Dropbox clone written in Go. | ||
|
||
The design is a client/server architecture using HTTP endpoints to communicate file change events between clients. A global journal of all file changes is kept on the server in a Postgres database. | ||
|
||
Local file changes are hashed and sent to the server, which discerns whether or not it has a file under that hash already. If not, the client uploads the file to the server, where the file is hashed to check for integrity, and if valid uploaded to an Amazon S3 instance. All other clients are alerted that a change has been made through a UDP socket, and then the other clients request the necessary changes through an HTTP endpoint. Clients then get the necessary changes directly from the Amazon S3 instance through an S3 signed URL. | ||
|
||
## Notes | ||
- Must have `GOBOX_AWS_ACCESS_KEY_ID` and `GOBOX_AWS_SECRET_ACCESS_KEY` set for aws client. | ||
- os.FileMode struct has all the information me need to handle files. symlink, permission, directory, etc.... | ||
- https://blogs.dropbox.com/tech/2014/07/streaming-file-synchronization/ | ||
- https://www.youtube.com/watch?v=PE4gwstWhmc | ||
|
||
## Api | ||
|
||
#### Server | ||
Endpoints: | ||
|
||
##### /meta: | ||
Recieves a piece of client metadata. Logs the request, and returns if the file needs to be uploaded to s3. | ||
```go | ||
resp, err := http.Post("/meta", "application/json", jsonMetadataBytes) | ||
// resp.Body #=> true | ||
``` | ||
|
||
##### /upload: | ||
Takes a file, computes a base256 hash value, uploads the file. Returns any errors or sucess. | ||
|
||
##### /download: | ||
Takes a file hash. (Confirms this user has access to that hash?) Passes the file from s3. | ||
|
||
##### /clients: | ||
Lists clients that this user has to check. | ||
|
||
##### /client/1?lastCheck=dateTime: | ||
Returns metadata for client by id after specified time. | ||
|
||
>**Notes:** | ||
>Does this work? Is this enough information for the filesystem to synch? | ||
>How do we store user data on the server? Are there any security issues? What if a new user wants to download all files? | ||
>The manifests are merely a log of all files changes. Can we combine the user manifests to create a server side record of the files that exist? Then show this to the user? Show this to new user clients? | ||
#### Server Endpoints: | ||
|
||
>How does the client parse the metadata and correctly download the right files? | ||
##### POST: /file-actions/ | ||
|
||
>Right now we're using lines of json in a file. How should we be storing it instead? Database? Would a database allow us to more easily create "computed" manifests? | ||
##### POST: /upload/ | ||
|
||
>Can we configure the meta endpoint to accept more than 1 piece of meta information? | ||
##### POST: /download/ | ||
|
||
##### POST: /clients/ | ||
|
||
## Resources | ||
- https://blogs.dropbox.com/tech/2014/07/streaming-file-synchronization/ |