Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Current issues with the dev version | Best way to support you bringing this forward? #9

Open
codiflow opened this issue May 24, 2024 · 9 comments

Comments

@codiflow
Copy link
Contributor

Hey @PatWie, I really appreciate your work and now the builds from the dev channel are working. 🥳

But there's still some work to do until the firebase storage is being replaced completely.

One thing that still seems to be missing is the storage of blobs (images and other uploads). They don't use firestore.googleapis.com but firebasestorage.googleapis.com and those requests are not being replaced currently.

I'm a DevOps guy and not a developer but I really would like to help you with bringing this forward. What is the best way to help you with this?

Thanks 😎
Christian

@codiflow
Copy link
Contributor Author

codiflow commented May 24, 2024

Now as I was playing around with your repo a bit these are my results:

My setup:

  • I was replacing locahost:3002 in main.go with a publicly resolvable hostname which is SSL terminated by nginx
  • I replaced "ssl=!0", "ssl=0" with "ssl=!0", "ssl=1" and "ssl:!0", "ssl:0" with "ssl:!0", "ssl:1" to be able to use HTTPS
  • I also set up a second setup using all the default values to cross-check the functionality

My findings:

  1. Collaboration is working out of the box without issues.
  2. ❌ The backend storage is not working (correctly) (neither in my SSL terminated setup nor in the default localhost setup). So you will always get the following error message after starting a collaboration session:

Couldn't save to the backend database. If problems persist, you should save your file locally to ensure you don't lose your work.

  1. ✅ Apart from that requests like the following are getting through:
http://localhost:3002/v1/projects/excalidraw-room-persistence/databases/(default)/documents:batchGet
  1. 💡 Using encryption functions in the browser is only possible with SSL encrypted connections. You will get errors in the browser console if that's not the case.

  2. ❌ I recognized that the following initial request is failing in both variants (SSL terminated and localhost). This is one of the requests where firebase.googleapis.com was replaced by localhost:3002:

http://localhost:3002/google.firestore.v1.Firestore/Listen/channel?database=projects%2Fexcalidraw-room-persistence%2Fdatabases%2F(default)&VER=8&RID=40837&CVER=22&X-HTTP-Session-Id=gsessionid&%24httpHeaders=X-Goog-Api-Client%3Agl-js%2F%20fire%2F8.3.3%0D%0AX-Firebase-GMPID%3A1%3A654800341332%3Aweb%3A4a692de832b55bd57ce0c1%0D%0AContent-Type%3Atext%2Fplain%0D%0A&zx=66dvi15g5r94&t=1

The chi middleware responds with error 404 here.

  1. ❌ The firebase storage firebasestorage.googleapis.com (which is being used for saving blobs like images) is not being hot-patched yet. You can try that on your own by uploading an image in Excalidraw and watch the browser logs (you have to be in a collaboration session during this test otherwise the image is only saved in the browsers storage).

This is an example request being sent while uploading a file:

OPTIONS https://firebasestorage.googleapis.com/v0/b//o?name=files/rooms/30f11559ba40ec4fc013/c48cb6746fe04d40064a3767d3033fec6d6777c9

POST https://firebasestorage.googleapis.com/v0/b//o?name=files%2Frooms%2F30f11559ba40ec4fc013%2Fc48cb6746fe04d40064a3767d3033fec6d6777c9

If I can be of any help with any of these issues please let me know 😎

@codiflow codiflow changed the title Best way to support you bringing this forward? Current issues with the dev version | Best way to support you bringing this forward? May 24, 2024
@PatWie
Copy link
Owner

PatWie commented May 25, 2024

I currently lack a clear understanding of which stubs or functionality we need to re-implement from the Firebase API. This could potentially be a significant time-sink. I still prefer not to modify any frontend code and instead focus on providing correct request/response handling for all fetches. My goal is to eventually enable easy self-hosting on AWS Fargate. However, I must admit, I am not super keen in extensive programming during my spare time at the moment.

The backend storage issue, i don't get. It works for me with link sharing using S3. Interestingly, I didn't know that one can embed arbitrary images 🙈. That's news to me.

@codiflow
Copy link
Contributor Author

codiflow commented May 27, 2024

I still prefer not to modify any frontend code and instead focus on providing correct request/response handling for all fetches.

Totally agree with that. Unfortunately I never used Firebase before so I can't provide any help either. Maybe I get a better understanding while diving deeper into the code.

What I can say is that there is a working (but outdated) fork out there where the upstream code was modified so a httpStorage backend can be used instead of Firebase. This is the repository: https://github.com/squatica/excalidraw

My goal is to eventually enable easy self-hosting on AWS Fargate. However, I must admit, I am not super keen in extensive programming during my spare time at the moment.

OK I won't use any Amazon services so this is where I'm out 😉 Anyway – I'm sure your approach is so great that we can evolve it so it's becoming an alternative to the current (code modifying) forks. Maybe I can find some other Go contributors who can help here 🙏

The backend storage issue, i don't get. It works for me with link sharing using S3.

Could you maybe try if the backend also works without S3? I'm using an sqlite db at the moment and will try it with the directory storage today.

Interestingly, I didn't know that one can embed arbitrary images 🙈. That's news to me.

Yes, that's one of the greatest features apart from already being a great piece of software at all 😎

@nhatsangvn
Copy link

nhatsangvn commented Jun 9, 2024

Hey @PatWie ,

This is great work, I really appreciated.

One issue that I got is even I set (before starting the ./excaldraw-complete)

export STORAGE_TYPE=sqlite
export DATA_SOURCE_NAME=/tmp/test.sqlite

the app seems doesn't use it. Although I can see /tmp/test.sqlite but when I draw some charts, that files doesn't change in size. Anyway that I can verify if the app actually use my sqlite db? (1)

Also is the the popup
image
relating to my STORAGE_TYPE above? (2)

PatWie added a commit that referenced this issue Jun 10, 2024
To troubleshoot reported issues, we add more debugging outputs in
the logging. This includes detailed, structured logs in the Create
and FindID methods to provide better traceability and debuggability.

These changes help diagnose issues by providing detailed context in
the logs, which should improve the ability to identify and resolve
problems.

See also: #9
@PatWie
Copy link
Owner

PatWie commented Jun 10, 2024

I added a commit with logging (and updated README.md). I verified S3, filesystem, memory and sqlite storage works here on Linux. Can you provide the logging output and OS info?

@nhatsangvn
Copy link

nhatsangvn commented Jun 10, 2024

Nevermind, I understand this feature. It only works if we use Save to > Sharable link.
Thank you @PatWie , I just re-tested it but the issue is still happened on my end.
My OS is Debian 12.
Log output:
~~ ~~root@snn-buil:/app# STORAGE_TYPE=sqlite DATA_SOURCE_NAME=test.db go run main.go --loglevel debug --listen=":3002"~~ ~~INFO[0000] Use storage dataSourceName=test.db storageType=sqlite~~ ~~INFO[0000] starting server addr=":3002"~~ ~~DEBU[0000] Server is running in the background~~ ~~2024/06/10 17:34:20 "GET http://mydomain/socket.io/?EIO=4&transport=websocket HTTP/1.1" from 127.0.0.1:33234 - 000 0B in 410.594µs~~ ~~init room %!(EXTRA socket.Room=iuEGYVNJd9S8HwAAAAAAAAAB)~~ ~~Socket iuEGYVNJd9S8HwAAAAAAAAAB has joined aa5a032c1897709ca992~~ ~~ room %!(EXTRA socket.Room=aa5a032c1897709ca992, string= has users , []socket.SocketId=[iuEGYVNJd9S8HwAAAAAAAAAB])~~ ~~ user iuEGYVNJd9S8HwAAAAAAAAAB sends volatile update to room aa5a032c1897709ca992~~ ~~ user iuEGYVNJd9S8HwAAAAAAAAAB sends volatile update to room aa5a032c1897709ca992~~ ~~ user iuEGYVNJd9S8HwAAAAAAAAAB sends volatile update to room aa5a032c1897709ca992~~ ~~ user iuEGYVNJd9S8HwAAAAAAAAAB sends volatile update to room aa5a032c1897709ca992~~ ~~ user iuEGYVNJd9S8HwAAAAAAAAAB sends volatile update to room aa5a032c1897709ca992~~ ~~...~~ ~~

~~This is the `.env.production` that I used to build the frontend:~~
~~```~~
~~VITE_APP_BACKEND_V2_GET_URL=https://mydomain/api/v2/~~
~~VITE_APP_BACKEND_V2_POST_URL=https://mydomain/api/v2/post/~~

VITE_APP_LIBRARY_URL=https://libraries.excalidraw.com
VITE_APP_LIBRARY_BACKEND=https://mydomain/libraries
VITE_APP_PLUS_LP=https://mydomain/plus/
VITE_APP_PLUS_APP=https://mydomain/app/

~~VITE_APP_AI_BACKEND=https://mydomain/ai/~~

VITE_APP_WS_SERVER_URL=https://mydomain

~~VITE_APP_FIREBASE_CONFIG='{"apiKey":"xxx-yyy","authDomain":"test-note.firebaseapp.com","databaseURL":"https://test-note.firebaseio.com","projectId":"test-note","storageBucket":"test-note.appspot.com","messagingSenderId":"201312225267","appId":"zzz"}'~~

VITE_APP_DISABLE_TRACKING=yes
```

~~When I started the server with~~
~~```~~
~~STORAGE_TYPE=sqlite DATA_SOURCE_NAME=test.db go run main.go --loglevel debug --listen=":3002"~~
~~```~~
~~I can see the `test.db` is there, but no new content is written.~~
~~<img width="1020" alt="image" src="https://github.com/PatWie/excalidraw-complete/assets/20770118/bf37c83c-1c90-4c1e-beec-a8f2eae4fbad">~~

After login to my domain (the app behind the reverse_proxy), open a live-collab, wait a few seconds and this issue happened.
image

~~I also notice that this request happened when I inspect the network:~~
~~<img width="1431" alt="image" src="https://github.com/PatWie/excalidraw-complete/assets/20770118/cbafd927-4c7c-4b20-b92e-ac1eaf7bf575">~~

May I know how did you test for the STORAGE_TYPE? Did you go to UI and hover around or which method?

@PatWie
Copy link
Owner

PatWie commented Jun 11, 2024

The issue in your screenshot is likely due to the required SSL for Firebase in the front-end implementation. One workaround is running Excalidraw locally using a reverse-proxy setup with Caddy. Caddy can automatically generate valid, self-signed SSL certificates and install them in the browser

@nhatsangvn
Copy link

Hi @PatWie , one new issue that I observed in the Room Server is:
image

when I try to open a colab room, sometimes the people in the rooms keep increasing although only 2 of my browsers are active.
image

Basically when I draw for sometimes (10s) and the number of people increases.

@PatWie
Copy link
Owner

PatWie commented Jul 6, 2024

This is unfortunately a known issue without a known root-cause:
#3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants