feat(dev): add devcontainers support

[uwla]

Development feature: automate the setup of development environment via devcontainers.

This facilitates developing Polar by configuring a reproducible environment to build, test and develop.

The devcontainers specification can be integrated into IDEs like Vscode/Vscodium.
Alternativaley, it is possible to manually create a container and run commands inside of it.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (0e8d29b) 100.00% compared to head (269e528) 100.00%.
Report is 12 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff            @@
##            master      #824   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files          139       139           
  Lines         4356      4389   +33     
  Branches       823       855   +32     
=========================================
+ Hits          4356      4389   +33     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jamaljsr]

Thanks for this PR. I have not played with devcontainers before. The concept looks fantastic. I see how this can be very helpful for new devs to get started on the project, so this update has my full support.

Unfortunately, I couldn't get this working on my machine. When I ran yarn inside of the devcontainer, it gave an error about make not found. Then when I ran yarn dev, the react dev server started up fine, but electron couldn't start.

View full logs

debian@eed5e49ee1e6:/workspaces/polar$ yarn
yarn install v1.22.19
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
warning " > @mrblenny/react-flow-chart@0.0.14" has incorrect peer dependency "react@^16.8.4".
warning " > @mrblenny/react-flow-chart@0.0.14" has incorrect peer dependency "react-dom@^16.8.4".
warning " > @mrblenny/react-flow-chart@0.0.14" has incorrect peer dependency "styled-components@^5.1.0".
warning "@mrblenny/react-flow-chart > react-zoom-pan-pinch@1.6.1" has unmet peer dependency "prop-types@^15.5.4".
warning "@mrblenny/react-flow-chart > react-zoom-pan-pinch@1.6.1" has incorrect peer dependency "react@^15.0.0 || ^16.0.0".
warning "@mrblenny/react-flow-chart > react-zoom-pan-pinch@1.6.1" has incorrect peer dependency "react-dom@^15.0.0 || ^16.0.0".
warning " > @testing-library/user-event@14.5.2" has unmet peer dependency "@testing-library/dom@>=7.21.4".
warning " > connected-react-router@6.9.3" has incorrect peer dependency "react-redux@^6.0.0 || ^7.1.0".
warning " > less-loader@11.1.4" has incorrect peer dependency "webpack@^5.0.0".
warning "react-lazylog > react-virtualized@9.21.2" has incorrect peer dependency "react@^15.3.0 || ^16.0.0-alpha".
warning "react-lazylog > react-virtualized@9.21.2" has incorrect peer dependency "react-dom@^15.3.0 || ^16.0.0-alpha".
warning " > react-router-transition@2.1.0" has incorrect peer dependency "react@^16.8.0".
warning " > react-router-transition@2.1.0" has incorrect peer dependency "react-dom@^16.8.0".
warning "react-router-transition > react-motion@0.5.2" has incorrect peer dependency "react@^0.14.9 || ^15.3.0 || ^16.0.0".
warning " > stylelint-config-prettier@9.0.5" has incorrect peer dependency "stylelint@>= 11.x < 15".
warning Workspaces can only be enabled in private projects.
[4/4] Building fresh packages...
[1/13] ⠄ protobufjs
[2/13] ⠄ protobufjs
[3/13] ⠄ cpu-features
[4/13] ⠄ core-js-pure
warning Error running install script for optional dependency: "/workspaces/polar/node_modules/cpu-features: Command failed.
Exit code: 1
Command: node buildcheck.js > buildcheck.gypi && node-gyp rebuild
Arguments: 
Directory: /workspaces/polar/node_modules/cpu-features
Output:
/workspaces/polar/node_modules/buildcheck/lib/index.js:115
  throw new Error('Unable to detect compiler type');
  ^

Error: Unable to detect compiler type
    at BuildEnvironment.getKind (/workspaces/polar/node_modules/buildcheck/lib/index.js:115:9)
    at BuildEnvironment.tryCompile (/workspaces/polar/node_modules/buildcheck/lib/index.js:537:15)
    at BuildEnvironment.checkHeader (/workspaces/polar/node_modules/buildcheck/lib/index.js:423:25)
    at Object.<anonymous> (/workspaces/polar/node_modules/cpu-features/buildcheck.js:16:4)
    at Module._compile (node:internal/modules/cjs/loader:1198:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1252:10)
    at Module.load (node:internal/modules/cjs/loader:1076:32)
[10/13] ⠐ core-js
[9/13] ⠐ dtrace-provider
[3/13] ⠐ cpu-features
[11/13] ⠐ utf-8-validate
error /workspaces/polar/node_modules/bufferutil: Command failed.
Exit code: 1
Command: node-gyp-build
Arguments: 
Directory: /workspaces/polar/node_modules/bufferutil
Output:
gyp info it worked if it ends with ok
gyp info using node-gyp@9.1.0
gyp info using node@16.20.2 | linux | arm64
gyp info find Python using Python version 3.11.2 found at "/usr/bin/python3"
gyp http GET https://nodejs.org/download/release/v16.20.2/node-v16.20.2-headers.tar.gz
gyp http 200 https://nodejs.org/download/release/v16.20.2/node-v16.20.2-headers.tar.gz
gyp http GET https://nodejs.org/download/release/v16.20.2/SHASUMS256.txt
gyp http 200 https://nodejs.org/download/release/v16.20.2/SHASUMS256.txt
gyp info spawn /usr/bin/python3
gyp info spawn args [
gyp info spawn args   '/home/debian/.nvm/versions/node/v16.20.2/lib/node_modules/npm/node_modules/node-gyp/gyp/gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'make',
gyp info spawn args   '-I',
gyp info spawn args   '/workspaces/polar/node_modules/bufferutil/build/config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/home/debian/.nvm/versions/node/v16.20.2/lib/node_modules/npm/node_modules/node-gyp/addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/home/debian/.cache/node-gyp/16.20.2/include/node/common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=/home/debian/.cache/node-gyp/16.20.2',
gyp info spawn args   '-Dnode_gyp_dir=/home/debian/.nvm/versions/node/v16.20.2/lib/node_modules/npm/node_modules/node-gyp',
gyp info spawn args   '-Dnode_lib_file=/home/debian/.cache/node-gyp/16.20.2/<(target_arch)/node.lib',
gyp info spawn args   '-Dmodule_root_dir=/workspaces/polar/node_modules/bufferutil',
gyp info spawn args   '-Dnode_engine=v8',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'build',
gyp info spawn args   '-Goutput_dir=.'
gyp info spawn args ]
gyp ERR! build error 
gyp ERR! stack Error: not found: make
gyp ERR! stack     at getNotFoundError (/home/debian/.nvm/versions/node/v16.20.2/lib/node_modules/npm/node_modules/which/which.js:10:17)
gyp ERR! stack     at /home/debian/.nvm/versions/node/v16.20.2/lib/node_modules/npm/node_modules/which/which.js:57:18
gyp ERR! stack     at new Promise (<anonymous>)
gyp ERR! stack     at step (/home/debian/.nvm/versions/node/v16.20.2/lib/node_modules/npm/node_modules/which/which.js:54:21)
gyp ERR! stack     at /home/debian/.nvm/versions/node/v16.20.2/lib/node_modules/npm/node_modules/which/which.js:71:22
gyp ERR! stack     at new Promise (<anonymous>)
gyp ERR! stack     at subStep (/home/debian/.nvm/versions/node/v16.20.2/lib/node_modules/npm/node_modules/which/which.js:69:33)
gyp ERR! stack     at /home/debian/.nvm/versions/node/v16.20.2/lib/node_modules/npm/node_modules/which/which.js:80:22
gyp ERR! stack     at /home/debian/.nvm/versions/node/v16.20.2/lib/node_modules/npm/node_modules/isexe/index.js:42:5
gyp ERR! stack     at /home/debian/.nvm/versions/node/v16.20.2/lib/node_modules/npm/node_modules/isexe/mode.js:8:5
gyp ERR! System Linux 6.4.16-linuxkit
gyp ERR! command "/home/debian/.nvm/versions/node/v16.20.2/bin/node" "/home/debian/.nvm/versions/node/v16.20.2/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /workspaces/polar/node_modules/bufferutil




debian@eed5e49ee1e6:/workspaces/polar$ yarn dev
yarn run v1.22.19
$ concurrently --kill-others --success first "yarn:dev:*"
$ yarn theme && cross-env BROWSER=none yarn start
$ wait-on http://localhost:3000 && nodemon -I --watch ./electron/ --watch ./src/shared/ --ext ts --exec electron ./public/dev.js
$ yarn theme:light && yarn theme:dark
$ lessc --include-path=node_modules --js ./src/theme/light.less ./public/themes/light.css
$ lessc --include-path=node_modules --js ./src/theme/dark.less ./public/themes/dark.css
$ rescripts start
[app] (node:10625) [DEP0148] DeprecationWarning: Use of deprecated folder mapping "./" in the "exports" field module resolution of the package at /workspaces/polar/node_modules/postcss-safe-parser/node_modules/postcss/package.json.
[app] Update this package.json to use a subpath pattern like "./*".
[app] (Use `node --trace-deprecation ...` to show where the warning was created)
[app] ℹ ｢wds｣: Project is running at http://172.17.0.3/
[app] ℹ ｢wds｣: webpack output is served from 
[app] ℹ ｢wds｣: Content not from webpack is served from /workspaces/polar/public
[app] ℹ ｢wds｣: 404s will fallback to /
[app] Starting the development server...
[app] 
[app] Browserslist: caniuse-lite is outdated. Please run:
[app] npx browserslist@latest --update-db
[app] 
[app] Why you should do it regularly:
[app] https://github.com/browserslist/browserslist#browsers-data-updating
[app] Browserslist: caniuse-lite is outdated. Please run the following command: `yarn upgrade`
[app] Compiled with warnings.
[app] 
[app] ./node_modules/ssh2/lib/protocol/crypto.js
[app] Module not found: Can't resolve './crypto/build/Release/sshcrypto.node' in '/workspaces/polar/node_modules/ssh2/lib/protocol'
[app] 
[app] ./node_modules/dtrace-provider/dtrace-provider.js
[app] Module not found: Can't resolve './src/build' in '/workspaces/polar/node_modules/dtrace-provider'
[app] 
[app] ./node_modules/ssh2/lib/protocol/constants.js
[app] Module not found: Can't resolve 'cpu-features' in '/workspaces/polar/node_modules/ssh2/lib/protocol'
[app] 
[app] Search for the keywords to learn more about each warning.
[app] To ignore, add // eslint-disable-next-line to the line before.
[app] 
[electron] [nodemon] 3.0.2
[electron] [nodemon] to restart at any time, enter `rs`
[electron] [nodemon] watching path(s): electron/**/* src/shared/**/*
[electron] [nodemon] watching extensions: ts
[electron] [nodemon] starting `electron ./public/dev.js`
[electron] /workspaces/polar/node_modules/electron/dist/electron: error while loading shared libraries: libgobject-2.0.so.0: cannot open shared object file: No such file or directory
[electron] [nodemon] failed to start process, "electron" exec not found
[electron] [nodemon] Error
[electron]     at Bus.<anonymous> (/workspaces/polar/node_modules/nodemon/lib/nodemon.js:158:25)
[electron]     at Bus.emit (node:events:525:35)
[electron]     at ChildProcess.<anonymous> (/workspaces/polar/node_modules/nodemon/lib/monitor/run.js:199:11)
[electron]     at ChildProcess.emit (node:events:513:28)
[electron]     at Process.ChildProcess._handle.onexit (node:internal/child_process:293:12)
[electron] yarn run dev:electron exited with code 0
--> Sending SIGTERM to other processes..
node:events:491
      throw er; // Unhandled 'error' event
      ^

Error: spawn ps ENOENT
    at Process.ChildProcess._handle.onexit (node:internal/child_process:285:19)
    at onErrorNT (node:internal/child_process:485:16)
    at processTicksAndRejections (node:internal/process/task_queues:83:21)
Emitted 'error' event on ChildProcess instance at:
    at Process.ChildProcess._handle.onexit (node:internal/child_process:291:12)
    at onErrorNT (node:internal/child_process:485:16)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'spawn ps',
  path: 'ps',
  spawnargs: [ '-o', 'pid', '--no-headers', '--ppid', 10233 ]
}
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

I can see how devcontainers are useful for web apps which just need to listen on a specified port, but do they also support electron-based desktop apps? I'm not sure how Polar can run inside of a docker container which has no UI. Can you provide some additional info on how you plan to use this with Polar?

[uwla]

Hi. I built a dev container and was able to install, test and compile polar inside the container, but I have not tried the access UI or hot-reload.

Here is an example:

1. Build the image:

docker image build --tag polardev:node16 .devcontainer/

2. Run a detached container "forever":

cd polar/
docker container run \
	--volume $PWD:/app \
	--name polar-dev \
	--detach \
	polardev:node16 \
	sleep infinity

3. Install packages:

docker exec polar-dev yarn install

4. Run linter:

docker exec polar-dev yarn lint

5. Run tests:

docker exec polar-dev yarn test

6. Build for the platform:

docker exec polar-dev yarn package

The examples above is to manually "use" devcontainers, but it does work in Vscode using the DevContainers extension. I could only get devcontainer to work in VSscode, but not Vscodium, the FLOSS fork.

Regarding the UI, I believe it is possible to forward the port from the container to the rust and access it via a standard browser. I also noticed I forgot to expose a port in the Dockerfile, so that needs to be addressed.

Let's work on this. Adding documentation will be helpful to future developers.

[jamaljsr]

Ah ok. Yes, I think we should have some comments in the CONTRIBUTING doc about how to use this.

[uwla]

Hi @jamaljsr, can you take a look again?
I think it should work now.

Also, I'd like to point out that I added the flag --no-sandbox in the dev:electron script. This is necessary in order to run this command inside the container, otherwise we would have to run the container with the privileged flag, which is riskier because bypass some of docker's security mechanisms. If the no-sandbox is not acceptable, I can remove it. But I think it should be acceptable because this is a development script.

[jamaljsr]

I just quickly skimmed through the diff. I'm shocked that you got the GUI working, I didn't think that would be possible.

I haven't tested but it seems like this would only work for linux systems since the getent and xhost commands aren't available on Mac & Windows. Do you have any insight on whether it's possible to get the UI working on those platforms as well? I have Mac & Windows machines I can test with. If not, I'm ok with merging this with only linux support. I'd just want to mention it in the instructions.

[uwla]

@jamaljsr running GUI applications using Docker is currently only possible using Linux distributions with X11 backend. It is not even possible in Linux Wayland. That's due to the flexibility of X11 protocol.

There is a project called x11docker, https://github.com/mviereck/x11docker, which automates running generic GUI applications in Docker, such as web brwosers, mail clients, file managers, video editors, and so on.

But it is possible to use the devcontainers in Windows and Mac in headless mode to install packages, lint code, run tests and compile Polar. I'll create a Docker-headless file and a Dockerfile-ui based on it.

Maybe, we should official polar development images. There is already polarlightning/bitcoin for example. We could have polarlightning/dev:headless-16 and polarlightning/dev:ui-16, where 16 denotes it is for node16. When Polar gets support for noed20, it could be polarlightning/dev:headless-20 and polarlightning/dev:ui-20.

Sounds good?

[jamaljsr]

@jamaljsr running GUI applications using Docker is currently only possible using Linux distributions with X11 backend. It is not even possible in Linux Wayland. That's due to the flexibility of X11 protocol.

Ok, thanks for the confirmation. Since this only works for Linux, I'd suggest adding a note in the instructions stating that. Also, I don't think these instructions should be at the top of the CONTRIBUTING doc since these instructions won't work for everyone. Can we keep the "Local approach" section first with these instructions below it. My thought is that the majority of people that want to contribute changes in the app (requiring the GUI to work), or they want to package a binary from source code to run on their system. I don't think many Mac/Windows users would go through this process just to lint, test, and build a Linux binary.

Maybe, we should official polar development images. There is already polarlightning/bitcoin for example. We could have polarlightning/dev:headless-16 and polarlightning/dev:ui-16, where 16 denotes it is for node16. When Polar gets support for noed20, it could be polarlightning/dev:headless-20 and polarlightning/dev:ui-20.

I like this idea. I can work on pushing these images to docker hub. Since the Node v20 update is now merged, I wouldn't do a v16 image since it won't work any longer.

[uwla]

That makes sense. I'll move the docs to the bottom of CONTRIBUTING and update node version to 20.

What is the minor version? I mean, is it 20.1 or 20.2, or something else.

[uwla]

@jamaljsr, I improved the PR

[jamaljsr]

LGTM 👌

I've pushed up images to Docker Hub. Unfortunately the ui image doesn't build for arm64, so I could only push that one for amd64. I attached the logs in case you have any suggestions. If not, we skip it.

Here's the logs with the error:
docker_build.log

Do you want to mention the Docker Hub images in the instructions before merging this?

[uwla]

I added mention to DockerHub images in the instructions.
This is a better approach: it is definetely faster to pull the images than to build them.

[jamaljsr]

Great! Thanks for the updates.