Skip to content

Commit

Permalink
Add different data providers and improve docs
Browse files Browse the repository at this point in the history
This adds support for switching between in-memory, local filesystem, s3
or sqlite.
  • Loading branch information
PatWie committed Mar 29, 2024
1 parent 2dd1421 commit 31c8029
Show file tree
Hide file tree
Showing 10 changed files with 426 additions and 89 deletions.
72 changes: 62 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,69 @@
# Exalidraw Complete
# Excalidraw Complete: A Self-Hosted Solution

Frustrated on how difficult it is to setup excalidraw self-hosted but with data
storage and collaboration function this represents and attempt to run the
necessary function with a single binary implemented in go. This includes:
Excalidraw Complete simplifies the deployment of Excalidraw, bringing an
all-in-one solution to self-hosting this versatile virtual whiteboard. Designed
for ease of setup and use, Excalidraw Complete integrates essential features
into a single Go binary. This solution encompasses:

- the frontend UI
- a in-memory data layer
- socket.io implementation for collaboration
- The intuitive Excalidraw frontend UI for seamless user experience.
- An integrated data layer ensuring fast and efficient data handling based on different data providers.
- A socket.io implementation to enable real-time collaboration among users.

The project goal is to alleviate the setup complexities traditionally associated with self-hosting Excalidraw, especially in scenarios requiring data persistence and collaborative functionalities.

## Installation

To get started, download the latest release binary:

```bash
# Visit https://github.com/PatWie/excalidraw-complete/releases/ for the download URL
wget <binary-download-url>
chmod +x excalidraw-complete
./excalidraw-complete
```

Once launched, Excalidraw Complete is accessible at `localhost:3002`, ready for
drawing and collaboration.

### Configuration

Excalidraw Complete adapts to your preferences with customizable storage solutions, adjustable via the `STORAGE_TYPE` environment variable:

- **Filesystem:** Opt for `STORAGE_TYPE=filesystem` and define `LOCAL_STORAGE_PATH` to use a local directory.
- **SQLite:** Select `STORAGE_TYPE=sqlite` with `DATA_SOURCE_NAME` for local SQLite storage, including the option for `:memory:` for ephemeral data.
- **AWS S3:** Choose `STORAGE_TYPE=s3` and specify `S3_BUCKET_NAME` to leverage S3 bucket storage, ideal for cloud-based solutions.

These flexible configurations ensure Excalidraw Complete fits seamlessly into your existing setup, whether on-premise or in the cloud.

## Building from Source

Interested in contributing or customizing? Build Excalidraw Complete from source with these steps:

```bash
# Clone and prepare the Excalidraw frontend
git clone https://github.com/excalidraw/excalidraw.git
cd excalidraw
git checkout tags/v0.17.3
git apply ../frontend.patch
# Follow build instructions to compile assets into the frontend directory
```

Compile the Go application:

Apply the patch to the frontend and build excalidraw into `frontend`. Run
```bash
go run main.go
go build -o excalidraw-complete main.go
```

Everything will be served under `localhost:3002`
Start the server:

```bash
./excalidraw-complete
```

Excalidraw Complete is now running on your machine, ready to bring your collaborative whiteboard ideas to life.

---

Excalidraw is a fantastic tool, but self-hosting it can be tricky. I welcome
your contributions to improve Excalidraw Complete — be it through adding new
features, improving existing ones, or bug reports.
19 changes: 19 additions & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,29 @@ require (
require (
github.com/ajg/form v1.5.1 // indirect
github.com/andybalholm/brotli v1.0.6 // indirect
github.com/aws/aws-sdk-go-v2 v1.26.0 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.1 // indirect
github.com/aws/aws-sdk-go-v2/config v1.27.9 // indirect
github.com/aws/aws-sdk-go-v2/credentials v1.17.9 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.0 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.4 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.4 // indirect
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.0 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.4 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.11.1 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.3.6 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.11.6 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.17.4 // indirect
github.com/aws/aws-sdk-go-v2/service/s3 v1.53.0 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.20.3 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.23.3 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.28.5 // indirect
github.com/aws/smithy-go v1.20.1 // indirect
github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572 // indirect
github.com/google/pprof v0.0.0-20230821062121-407c9e7a662f // indirect
github.com/gookit/color v1.5.4 // indirect
github.com/gorilla/websocket v1.5.0 // indirect
github.com/mattn/go-sqlite3 v1.14.22 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect
github.com/onsi/ginkgo/v2 v2.12.0 // indirect
github.com/quic-go/qpack v0.4.0 // indirect
Expand Down
38 changes: 38 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,42 @@ github.com/ajg/form v1.5.1 h1:t9c7v8JUKu/XxOGBU0yjNpaMloxGEJhUkqFRq0ibGeU=
github.com/ajg/form v1.5.1/go.mod h1:uL1WgH+h2mgNtvBq0339dVnzXdBETtL2LeUXaIv25UY=
github.com/andybalholm/brotli v1.0.6 h1:Yf9fFpf49Zrxb9NlQaluyE92/+X7UVHlhMNJN2sxfOI=
github.com/andybalholm/brotli v1.0.6/go.mod h1:fO7iG3H7G2nSZ7m0zPUDn85XEX2GTukHGRSepvi9Eig=
github.com/aws/aws-sdk-go-v2 v1.26.0 h1:/Ce4OCiM3EkpW7Y+xUnfAFpchU78K7/Ug01sZni9PgA=
github.com/aws/aws-sdk-go-v2 v1.26.0/go.mod h1:35hUlJVYd+M++iLI3ALmVwMOyRYMmRqUXpTtRGW+K9I=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.1 h1:gTK2uhtAPtFcdRRJilZPx8uJLL2J85xK11nKtWL0wfU=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.1/go.mod h1:sxpLb+nZk7tIfCWChfd+h4QwHNUR57d8hA1cleTkjJo=
github.com/aws/aws-sdk-go-v2/config v1.27.9 h1:gRx/NwpNEFSk+yQlgmk1bmxxvQ5TyJ76CWXs9XScTqg=
github.com/aws/aws-sdk-go-v2/config v1.27.9/go.mod h1:dK1FQfpwpql83kbD873E9vz4FyAxuJtR22wzoXn3qq0=
github.com/aws/aws-sdk-go-v2/credentials v1.17.9 h1:N8s0/7yW+h8qR8WaRlPQeJ6czVMNQVNtNdUqf6cItao=
github.com/aws/aws-sdk-go-v2/credentials v1.17.9/go.mod h1:446YhIdmSV0Jf/SLafGZalQo+xr2iw7/fzXGDPTU1yQ=
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.0 h1:af5YzcLf80tv4Em4jWVD75lpnOHSBkPUZxZfGkrI3HI=
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.0/go.mod h1:nQ3how7DMnFMWiU1SpECohgC82fpn4cKZ875NDMmwtA=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.4 h1:0ScVK/4qZ8CIW0k8jOeFVsyS/sAiXpYxRBLolMkuLQM=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.4/go.mod h1:84KyjNZdHC6QZW08nfHI6yZgPd+qRgaWcYsyLUo3QY8=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.4 h1:sHmMWWX5E7guWEFQ9SVo6A3S4xpPrWnd77a6y4WM6PU=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.4/go.mod h1:WjpDrhWisWOIoS9n3nk67A3Ll1vfULJ9Kq6h29HTD48=
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.0 h1:hT8rVHwugYE2lEfdFE0QWVo81lF7jMrYJVDWI+f+VxU=
github.com/aws/aws-sdk-go-v2/internal/ini v1.8.0/go.mod h1:8tu/lYfQfFe6IGnaOdrpVgEL2IrrDOf6/m9RQum4NkY=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.4 h1:SIkD6T4zGQ+1YIit22wi37CGNkrE7mXV1vNA5VpI3TI=
github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.4/go.mod h1:XfeqbsG0HNedNs0GT+ju4Bs+pFAwsrlzcRdMvdNVf5s=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.11.1 h1:EyBZibRTVAs6ECHZOw5/wlylS9OcTzwyjeQMudmREjE=
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.11.1/go.mod h1:JKpmtYhhPs7D97NL/ltqz7yCkERFW5dOlHyVl66ZYF8=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.3.6 h1:NkHCgg0Ck86c5PTOzBZ0JRccI51suJDg5lgFtxBu1ek=
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.3.6/go.mod h1:mjTpxjC8v4SeINTngrnKFgm2QUi+Jm+etTbCxh8W4uU=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.11.6 h1:b+E7zIUHMmcB4Dckjpkapoy47W6C9QBv/zoUP+Hn8Kc=
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.11.6/go.mod h1:S2fNV0rxrP78NhPbCZeQgY8H9jdDMeGtwcfZIRxzBqU=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.17.4 h1:uDj2K47EM1reAYU9jVlQ1M5YENI1u6a/TxJpf6AeOLA=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.17.4/go.mod h1:XKCODf4RKHppc96c2EZBGV/oCUC7OClxAo2MEyg4pIk=
github.com/aws/aws-sdk-go-v2/service/s3 v1.53.0 h1:r3o2YsgW9zRcIP3Q0WCmttFVhTuugeKIvT5z9xDspc0=
github.com/aws/aws-sdk-go-v2/service/s3 v1.53.0/go.mod h1:w2E4f8PUfNtyjfL6Iu+mWI96FGttE03z3UdNcUEC4tA=
github.com/aws/aws-sdk-go-v2/service/sso v1.20.3 h1:mnbuWHOcM70/OFUlZZ5rcdfA8PflGXXiefU/O+1S3+8=
github.com/aws/aws-sdk-go-v2/service/sso v1.20.3/go.mod h1:5HFu51Elk+4oRBZVxmHrSds5jFXmFj8C3w7DVF2gnrs=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.23.3 h1:uLq0BKatTmDzWa/Nu4WO0M1AaQDaPpwTKAeByEc6WFM=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.23.3/go.mod h1:b+qdhjnxj8GSR6t5YfphOffeoQSQ1KmpoVVuBn+PWxs=
github.com/aws/aws-sdk-go-v2/service/sts v1.28.5 h1:J/PpTf/hllOjx8Xu9DMflff3FajfLxqM5+tepvVXmxg=
github.com/aws/aws-sdk-go-v2/service/sts v1.28.5/go.mod h1:0ih0Z83YDH/QeQ6Ori2yGE2XvWYv/Xm+cZc01LC6oK0=
github.com/aws/smithy-go v1.20.1 h1:4SZlSlMr36UEqC7XOyRVb27XMeZubNcBNN+9IgEPIQw=
github.com/aws/smithy-go v1.20.1/go.mod h1:krry+ya/rV9RDcV/Q16kpu6ypI4K2czasz0NC3qS14E=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
Expand All @@ -27,6 +63,8 @@ github.com/gookit/color v1.5.4 h1:FZmqs7XOyGgCAxmWyPslpiok1k05wmY3SJTytgvYFs0=
github.com/gookit/color v1.5.4/go.mod h1:pZJOeOS8DM43rXbp4AZo1n9zCU2qjpcRko0b6/QJi9w=
github.com/gorilla/websocket v1.5.0 h1:PPwGk2jz7EePpoHN/+ClbZu8SPxiqlu12wZP/3sWmnc=
github.com/gorilla/websocket v1.5.0/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
github.com/mattn/go-sqlite3 v1.14.22 h1:2gZY6PC6kBnID23Tichd1K+Z0oS6nE/XwU+Vz/5o4kU=
github.com/mattn/go-sqlite3 v1.14.22/go.mod h1:Uh1q+B4BYcTPb+yiD3kU8Ct7aC0hY9fxUwlHK0RXw+Y=
github.com/mitchellh/mapstructure v1.5.0 h1:jeMsZIYE/09sWLaz43PL7Gy6RuMjD2eJVyuac5Z2hdY=
github.com/mitchellh/mapstructure v1.5.0/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo=
github.com/oklog/ulid/v2 v2.1.0 h1:+9lhoxAP56we25tyYETBBY1YLA2SaoLvUFgrP2miPJU=
Expand Down
48 changes: 48 additions & 0 deletions handlers/api/documents.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
package documents

import (
"bytes"
"excalidraw-complete/core"
"io"
"net/http"

"github.com/go-chi/chi/v5"
"github.com/go-chi/render"
)

type (
DocumentCreateResponse struct {
ID string `json:"id"`
}
)

func HandleCreate(documentStore core.DocumentStore) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
data := new(bytes.Buffer)
_, err := io.Copy(data, r.Body)
if err != nil {
http.Error(w, "Failed to copy", http.StatusInternalServerError)
return
}
id, err := documentStore.Create(r.Context(), &core.Document{Data: *data})
if err != nil {
http.Error(w, "Failed to save", http.StatusInternalServerError)
return
}

render.JSON(w, r, DocumentCreateResponse{ID: id})
render.Status(r, http.StatusOK)
}
}

func HandleGet(documentStore core.DocumentStore) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "id")
document, err := documentStore.FindID(r.Context(), id)
if err != nil {
http.Error(w, "not found", http.StatusNotFound)
return
}
w.Write(document.Data.Bytes())
}
}
123 changes: 44 additions & 79 deletions main.go
Original file line number Diff line number Diff line change
@@ -1,34 +1,26 @@
package main

import (
"bytes"
"embed"
_ "embed"
"excalidraw-complete/core"
"excalidraw-complete/documents/memory"
documents "excalidraw-complete/handlers/api"
"excalidraw-complete/stores"
"fmt"
"io"
"io/fs"
"net/http"
"os"
"os/signal"
"path"
"strings"
"syscall"

"github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware"
"github.com/go-chi/cors"
"github.com/go-chi/render"
"github.com/zishang520/engine.io/v2/types"
socketio "github.com/zishang520/socket.io/v2/socket"
)

type (
DocumentCreateResponse struct {
ID string `json:"id"`
}

UserToFollow struct {
SocketId string
Username string
Expand All @@ -42,33 +34,35 @@ type (
//go:embed all:frontend
var assets embed.FS

func Assets() (fs.FS, error) {
return fs.Sub(assets, "frontend")
}

type FrontEndHandler struct{}

func (h FrontEndHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
frontendHandler(w, r)
}

func frontendHandler(w http.ResponseWriter, r *http.Request) {
upath := r.URL.Path
if !strings.HasPrefix(upath, "/") {
upath = "/" + upath
r.URL.Path = upath
}
upath = path.Clean(upath)

func handleUI() http.Handler {
sub, err := fs.Sub(assets, "frontend")
if err != nil {
panic(err)
}
http.FileServer(http.FS(sub)).ServeHTTP(w, r)

return http.FileServer(http.FS(sub))
}

func main() {
func setupRouter(documentStore core.DocumentStore) *chi.Mux {
r := chi.NewRouter()
r.Use(middleware.Logger)

r.Use(cors.Handler(cors.Options{
AllowedOrigins: []string{"https://*", "http://*"},
AllowedMethods: []string{"GET", "POST", "PUT", "DELETE", "OPTIONS"},
AllowedHeaders: []string{"Accept", "Authorization", "Content-Type", "Content-Length", "X-CSRF-Token", "Token", "session", "Origin", "Host", "Connection", "Accept-Encoding", "Accept-Language", "X-Requested-With"},
AllowCredentials: true,
MaxAge: 300, // Maximum value not ignored by any of major browsers
}))

r.Route("/api/v2", func(r chi.Router) {
r.Post("/post/", documents.HandleCreate(documentStore))
r.Route("/{id}", func(r chi.Router) {
r.Get("/", documents.HandleGet(documentStore))
})
})
return r
}
func setupSocketIO() *socketio.Server {
opts := socketio.DefaultServerOptions()
opts.SetMaxHttpBufferSize(5000000)
opts.SetPath("/socket.io")
Expand Down Expand Up @@ -153,56 +147,11 @@ func main() {
socket.Disconnect(true)
})
})
return ioo

r := chi.NewRouter()
r.Use(middleware.Logger)

r.Use(cors.Handler(cors.Options{
AllowedOrigins: []string{"https://*", "http://*"},
AllowedMethods: []string{"GET", "POST", "PUT", "DELETE", "OPTIONS"},
AllowedHeaders: []string{"Accept", "Authorization", "Content-Type", "Content-Length", "X-CSRF-Token", "Token", "session", "Origin", "Host", "Connection", "Accept-Encoding", "Accept-Language", "X-Requested-With"},
AllowCredentials: true,
MaxAge: 300, // Maximum value not ignored by any of major browsers
}))

documentStore := memory.NewDocumentStore()

r.Mount("/", FrontEndHandler{})

r.Route("/api/v2", func(r chi.Router) {
r.Post("/post/", func(w http.ResponseWriter, r *http.Request) {
data := new(bytes.Buffer)
_, err := io.Copy(data, r.Body)
if err != nil {
http.Error(w, "Failed to copy", http.StatusInternalServerError)
return
}
id, err := documentStore.Create(r.Context(), &core.Document{Data: *data})
if err != nil {
http.Error(w, "Failed to save", http.StatusInternalServerError)
return
}

render.JSON(w, r, DocumentCreateResponse{ID: id})
render.Status(r, http.StatusOK)
})
r.Route("/{id}", func(r chi.Router) {
r.Get("/", func(w http.ResponseWriter, r *http.Request) {
id := chi.URLParam(r, "id")
document, err := documentStore.FindID(r.Context(), id)
if err != nil {
http.Error(w, "not found", http.StatusNotFound)
return
}
w.Write(document.Data.Bytes())
})
})
})

r.Handle("/socket.io/", ioo.ServeHandler(nil))
go http.ListenAndServe(":3002", r)
fmt.Println("listen on 3002")
}

func waitForShutdown(ioo *socketio.Server) {
exit := make(chan struct{})
SignalC := make(chan os.Signal)

Expand All @@ -220,4 +169,20 @@ func main() {
<-exit
ioo.Close(nil)
os.Exit(0)
fmt.Println("Shutting down...")
// TODO(patwie): Close other resources
os.Exit(0)
}

func main() {
documentStore := stores.GetStore() // Make sure this is well-defined in your "stores" package
r := setupRouter(documentStore)
ioo := setupSocketIO()
r.Handle("/socket.io/", ioo.ServeHandler(nil))
r.Mount("/", handleUI())

go http.ListenAndServe(":3002", r)
fmt.Println("listen on 3002")
waitForShutdown(ioo)

}
Loading

0 comments on commit 31c8029

Please sign in to comment.