Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

self-hosted version #115

Closed
sungwoncho opened this issue Sep 14, 2018 · 14 comments
Closed

self-hosted version #115

sungwoncho opened this issue Sep 14, 2018 · 14 comments

Comments

@sungwoncho
Copy link
Member

@sungwoncho sungwoncho commented Sep 14, 2018

Just an idea:

  • allow users to download Dnote server and self-host
  • allow users to download Dnote web app and self-host
  • allow users to configure backup url to point to the self-hosted server
  • continue to provide a hosted option as a paid offering

question:

  • should the Dnote server and web version be:
    • fully open source
    • an open source community edition with limited features
    • a one time purchase
  • if flat fee, how would the server binary and web app be distributed with proper access control for users who purchased the software?
  • any other ideas?

IDEA:

  • open source the server for syncing notes
@sungwoncho
Copy link
Member Author

@sungwoncho sungwoncho commented May 10, 2019

#171 open sourced the the server side code, and now we are one step closer to self-hosting. Some things that we need to think about before formalizing self-host guideline:

  • a reliable way for users to run SQL migration files that need to be run.
  • a simple script to set up and run Dnote
  • docker?
  • how would users apply new patch from master?

EDIT

  • In CLI's .dnoterc, allow to specify the endpoint for sync.

Loading

@pietvanzoen
Copy link

@pietvanzoen pietvanzoen commented May 10, 2019

Excellent! Thanks for the work on this.

For my purposes I would be running this from a docker image. Could be something you setup and publish automatically from your main branch build pipeline.

Loading

@aravindhsampath
Copy link

@aravindhsampath aravindhsampath commented May 13, 2019

Just learned about Dnote. It sounds very promising and would fit my needs well. Just wanted share some thoughts.

  1. Being built with Go, it would be lovely to get simple executable binaries as an option of deployment. Updating dnote? - Just download the latest released binary and put it in place.
  2. Have not yet checked on the SQL side of things - whether dnote needs a full blown database or a simple SQLite. A simple script that would set it up would be a good idea of simple automation.
  3. I like a good old fashioned test/QA release. Whenever you think a feature is complete, you can release a build that is marked for testing, and folks who want to visit the bleeding edge can test and report issues. When you deem it ready, you can promote it to a stable release.
  4. Docker? - I suppose there is a wider audience that would prefer to just "docker run" something. Just wanted to raise my hands here and say that there are odd ones like me who likes to inspect what I'm running and not have it run by a complex daemon running with root privileges. Also, places like OpenBSD where Docker is not commonplace.

Cheers on building such a good project!!!!!

Loading

@sungwoncho
Copy link
Member Author

@sungwoncho sungwoncho commented May 14, 2019

@aravindhsampath Thanks for your input. Where did you learn about Dnote?

  1. It's possible to just provide binaries. But for web client, users will still need to download the latest JavaScript bundle and assets.
  2. Locally Dnote uses SQLite, and there is a module that runs any pending migration. The server side uses postgres and needs a way to automate migration. A bash script can do the work, but users might forget to run it, if not automated.
  3. That is a pretty cool idea akin to FireFox's nightly release. But I will keep it simple for now by just having a master branch as a stable release.
  4. Yeah, we could have a simple bash script to set up and run Dnote, instead of using docker. Could you explain the meaning of "have it run by a complex daemon running with root privileges"? Isn't it possible to run docker as non-root?

Loading

@sungwoncho
Copy link
Member Author

@sungwoncho sungwoncho commented Jul 25, 2019

I have done an initial release for Dnote server. It is a single binary that encompasses the front-end app, API server, and background jobs. Upgrading to new version is as simple as downloading and running a new binary. (thanks for your feedback @aravindhsampath) Database migration is also automated.

There is an official self hosting guide. Currently, Dnote server has the same feature you see on production site. It is marked as a beta release only to allow more improvement on self hosting guide before a stable release.

  • Would it make sense to automate setting up postgres, nginx, systemd by creating an installation script? How feasible is it to do so in a cross-platform manner at least for *nix systems? Such script can be helpful for setting up LetsEncrypt as well.
  • Any other feedback welcomed.

Loading

@CrCs2O4
Copy link

@CrCs2O4 CrCs2O4 commented Sep 3, 2019

Hi @sungwoncho!

Thanks a lot for a project 👍 I spined up self-hosted version locally to try (my simple docker-compose env).

Can you elaborate more how to connect CLI ?

In CLI's .dnoterc, allow to specify the endpoint for sync

I installed CLI with brew install dnote, so where I need to put .dnoterc and what variables to put inside? Can you provide example config?

Thanks,
Mike.

Loading

@sungwoncho
Copy link
Member Author

@sungwoncho sungwoncho commented Sep 7, 2019

@CrCs2O4 .dnoterc is generated under ~/.dnote upon running the CLI the first time. For instance, in my machine, it looks like:

▶ cat ~/.dnote/dnoterc 
editor: nvim
apiEndpoint: https://api.dnote.io

Please configure the value for apiEndpoint to the URL to your self hosted instance. Let me know if it worked for you. Thanks for trying Dnote.

Loading

@CrCs2O4
Copy link

@CrCs2O4 CrCs2O4 commented Sep 25, 2019

thanks for an answer! I configured apiEndpoint in ~/.dnote but when I tried to login dnote login I have an error:

logging in: getting presiginin: decoding payload: invalid character '<' looking for beginning of value

I installed proxy inside docker-compose to see the requests going to dnote-server and there is only one:

proxy_1     | 172.21.0.1 - - [18/Sep/2019:15:14:22 +0000] "GET /v1/presignin?email=test@test.test HTTP/1.1" 200 3795 "-" "-" 42 "dnote@docker" "http://172.21.0.5:3000" 0ms

Smth is wrong 😃 is there a way to run dnote client/server in debug mode to see logs?

Loading

@sungwoncho
Copy link
Member Author

@sungwoncho sungwoncho commented Sep 26, 2019

@CrCs2O4 From the error message, the problem could be that the server is returning HTML instead of JSON, possibly due to misconfigured proxy.

is there a way to run dnote client/server in debug mode to see logs?

Please upgrade the CLI to v0.9.1 which contains #252 to help you debug. You can see some logs for HTTP requests by setting the environment variable DNOTE_DEBUG=1 while running dnote.

Please try DNOTE_DEBUG=1 dnote login and look for the output such as the following:

DEBUG: HTTP request: &{Method:GET URL:https://api.dnote.io/v1/presignin?email=a Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Cli-Version:[master]] Body:{} GetBody:0x77dc40 ContentLength:0 TransferEncoding:[] Close:false Host:api.dnote.io Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr: RequestURI: TLS:<nil> Cancel:<nil> Response:<nil> ctx:0xc000024150}

And please check that the URL in the log is correct, and see if that URL returns a correct JSON payload by pinging it manually using tools such as curl.

Loading

@CrCs2O4
Copy link

@CrCs2O4 CrCs2O4 commented Sep 26, 2019

thanks for a quick responce! I assume it's not misconfigured proxy, because I can login in browser, create notes and everything works.

Debug log form cli:

DEBUG: initializing the database
DEBUG: initializing the system
DEBUG: current schema: schema 12 of 12
DEBUG: Running with Dnote context: {HomeDir:~ DnoteDir:~/.dnote APIEndpoint:http://dnote.docker.localhost Version:0.9.1 DB:0xc00000e8c0 SessionKey:0 SessionKeyExpiry:0 CipherKey:[1] Editor:vi Clock:0x4c3ff10}
  Welcome to Dnote Pro (https://dnote.io).
  [?] email: test@test.test
  [?] password:
DEBUG: HTTP request: &{Method:GET URL:http://dnote.docker.localhost/v1/presignin?email=test@test.test Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Cli-Version:[0.9.1]] Body:{} GetBody:0x435fa00 ContentLength:0 TransferEncoding:[] Close:false Host:dnote.docker.localhost Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr: RequestURI: TLS:<nil> Cancel:<nil> Response:<nil> ctx:<nil>}
  ⨯ logging in: getting presiginin: decoding payload: invalid character '<' looking for beginning of value

On server side there is only one request:

proxy_1     | 172.26.0.1 - - [26/Sep/2019:09:14:58 +0000] "GET /v1/presignin?email=test@test.test HTTP/1.1" 200 3795 "-" "-" 41 "dnote@docker" "http://172.26.0.5:3000" 1ms

And when I'm login in from browser its much more:

dnote_1     | 2019/09/26 09:13:04 GET	/api/v1/presignin?email=test@test.test	801.8µs
proxy_1     | 172.26.0.1 - - [26/Sep/2019:09:13:04 +0000] "GET /api/v1/presignin?email=test@test.test HTTP/1.1" 200 21 "-" "-" 34 "dnote@docker" "http://172.26.0.5:3000" 2ms
dnote_1     | 2019/09/26 09:13:05 POST	/api/v1/signin	102.1189ms
proxy_1     | 172.26.0.1 - - [26/Sep/2019:09:13:05 +0000] "POST /api/v1/signin HTTP/1.1" 200 179 "-" "-" 35 "dnote@docker" "http://172.26.0.5:3000" 103ms
dnote_1     | 2019/09/26 09:13:05 account is {Model:{ID:1 CreatedAt:2019-09-26 09:09:21.849472 +0000 UTC UpdatedAt:2019-09-26 09:09:21.849472 +0000 UTC} UserID:1 AccountID: Nickname: Provider: Email:{NullString:{String:test@test.test Valid:true}} EmailVerified:false Password:{NullString:{String: Valid:false}} ClientKDFIteration:100000 ServerKDFIteration:100000 AuthKeyHash:+aMJccQwTyZtxOB1GvIY0BH/mNURfGTgHjM3nPpDIFY= Salt:YwUXGyDmxac4255Dsoz/Mg== CipherKeyEnc:zxisIfp3BHLa8VPl36b2T81BCl0kiBdgBCFn124cs8ScvqYYEjhYG0OedpW2dNaZaGIpV8PWywu1J8PQ}
dnote_1     | 2019/09/26 09:13:05 GET	/api/me	29.9497ms
proxy_1     | 172.26.0.1 - - [26/Sep/2019:09:13:05 +0000] "GET /api/me HTTP/1.1" 200 289 "-" "-" 36 "dnote@docker" "http://172.26.0.5:3000" 34ms
dnote_1     |
dnote_1     | (/home/sung/dev/go/src/github.com/dnote/dnote/pkg/server/api/handlers/notes.go:155)
dnote_1     | [2019-09-26 09:13:05]  [2.92ms]  SELECT count(*) FROM "notes"  WHERE (notes.user_id = 1 AND notes.deleted = false) AND (notes.added_on >= 1567296000000000000 AND notes.added_on < 1569888000000000000)

Maybe it's ctx:<nil> nil value, that tries to be parsed and produces error decoding payload: invalid character '<' looking for beginning of value ?

Loading

@sungwoncho
Copy link
Member Author

@sungwoncho sungwoncho commented Sep 26, 2019

@CrCs2O4 Thanks for sharing the debug information.

From the output, it looks like your apiEndpoint value is set to http://dnote.docker.localhost. Could you try to set it to http://dnote.docker.localhost/api?

Loading

@CrCs2O4
Copy link

@CrCs2O4 CrCs2O4 commented Sep 27, 2019

and now it works and I can start feeling stupid 😆

DEBUG: initializing the database
DEBUG: initializing the system
DEBUG: current schema: schema 12 of 12
DEBUG: Running with Dnote context: {HomeDir:~/ DnoteDir:~/.dnote APIEndpoint:http://dnote.docker.localhost/api Version:0.9.1 DB:0xc0000b0840 SessionKey:0 SessionKeyExpiry:0 CipherKey:[1] Editor:vi Clock:0x4c3ff10}
  Welcome to Dnote Pro (https://dnote.io).
  [?] email: test@test.test
  [?] password:
DEBUG: HTTP request: &{Method:GET URL:http://dnote.docker.localhost/api/v1/presignin?email=test@test.test Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Cli-Version:[0.9.1]] Body:{} GetBody:0x435fa00 ContentLength:0 TransferEncoding:[] Close:false Host:dnote.docker.localhost Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr: RequestURI: TLS:<nil> Cancel:<nil> Response:<nil> ctx:<nil>}
DEBUG: HTTP request: &{Method:POST URL:http://dnote.docker.localhost/api/v1/signin Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Cli-Version:[0.9.1]] Body:{Reader:0xc0000b0b80} GetBody:0x435f920 ContentLength:84 TransferEncoding:[] Close:false Host:dnote.docker.localhost Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr: RequestURI: TLS:<nil> Cancel:<nil> Response:<nil> ctx:<nil>}
  ✔ logged in

Thanks a lot for help! 👍

Loading

@sungwoncho
Copy link
Member Author

@sungwoncho sungwoncho commented Sep 27, 2019

@CrCs2O4 No worries. I have updated the documentation in #253.

Loading

@sungwoncho
Copy link
Member Author

@sungwoncho sungwoncho commented Nov 4, 2019

Self-hosted version is out of beta and I will close this. A discussion will continue about how to make self-hosting easier on #275.

Loading

@sungwoncho sungwoncho closed this Nov 4, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants