Our element-desktop is not built with encrypted chat search support

[colemickens]

Describe the bug

1. Open Riot-Desktop
2. Settings -> Security & Privacy

Under the "Message search" heading, observe:

Riot is missing some components required for securely caching encrypted messages locally. If you'd like to experiment with this feature, build a custom Riot Desktop with search components added.

it links to here: https://github.com/vector-im/riot-web/blob/develop/docs/native-node-modules.md#adding-seshat-for-search-in-e2e-encrypted-rooms

Notify maintainers
cc: @pacien @worldofpeace

Metadata

 - system: `"x86_64-linux"`
 - host os: `Linux 5.6.12, NixOS, 20.09pre-git (Nightingale)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.4`
 - nixpkgs: `/home/cole/code/nixpkgs/cmpkgs`

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute: riot-desktop

[livnev]

Would be fantastic to have riot-desktop built with seshat. I tried building it myself but couldn't do it. 😢

[vbrandl]

Just stumbled across this myself. What's the current state on this? What needs to be done before this can be fixed?

[ajs124]

So, element seems to use its own build system hak to build these native dependencies. It tries to download and build those itself with cargo and stuff. Seems quite complicated to package.

[sagehane]

I would like to add that support for desktop notifications would be also appreciated. (They showed up on Arch's version, at least)

Edit: This problem has been resolved since

[ghost]

I made some progress. This solves the issue of downloading the hakDependencies. Unfortunately they are not locked in any way, so we need to lock them at packaging-time (is that even a term?) by adding them to yarn.lock and yarn.nix. Since yarn2nix always runs yarn with --ignore-scripts, we can safely ignore the upstream reasoning for fetching them outside of yarn dependencies.

I'm now at the point where it tries to build seshat in the element-desktop derivation, but unfortunately they don't provide a Cargo lockfile. I'm working on resolving this problem upstream:
matrix-org/seshat#79

diff --git a/pkgs/applications/networking/instant-messengers/element/element-desktop.nix b/pkgs/applications/networking/instant-messengers/element/element-desktop.nix
index 332563476d1..c73000234e4 100644
--- a/pkgs/applications/networking/instant-messengers/element/element-desktop.nix
+++ b/pkgs/applications/networking/instant-messengers/element/element-desktop.nix
@@ -1,6 +1,7 @@
 { lib, stdenv, fetchFromGitHub
 , makeWrapper, makeDesktopItem, mkYarnPackage
 , electron, element-web
+, python
 }:
 # Notes for maintainers:
 # * versions of `element-web` and `element-desktop` should be kept in sync.
@@ -20,9 +21,31 @@ in mkYarnPackage rec {
   inherit version src;

   packageJSON = ./element-desktop-package.json;
+  yarnLock = ./element-desktop-yarndeps.lock;
   yarnNix = ./element-desktop-yarndeps.nix;

-  nativeBuildInputs = [ makeWrapper ];
+  nativeBuildInputs = [ makeWrapper python ];
+
+  postConfigure = ''
+    rm deps/element-desktop/node_modules
+    cp -R "$node_modules" deps/element-desktop
+    chmod -R u+w deps/element-desktop
+  '';
+
+  preBuild = ''
+    (
+      cd deps/element-desktop
+      mkdir -p .hak/matrix-seshat
+      ln -s $PWD/node_modules/matrix-seshat .hak/matrix-seshat/build
+      mkdir -p .hak/keytar
+      ln -s $PWD/node_modules/keytar .hak/keytar/build
+      node scripts/hak/index.js check
+      node scripts/hak/index.js build
+    )
+  '';

   installPhase = ''
     # resources
diff --git a/pkgs/applications/networking/instant-messengers/element/update-element-desktop.sh b/pkgs/applications/networking/instant-messengers/element/update-element-desktop.sh
index 69d0d3d7072..4638997fc9b 100755
--- a/pkgs/applications/networking/instant-messengers/element/update-element-desktop.sh
+++ b/pkgs/applications/networking/instant-messengers/element/update-element-desktop.sh
@@ -1,5 +1,5 @@
 #!/usr/bin/env nix-shell
-#!nix-shell -I nixpkgs=../../../../../ -i bash -p wget yarn2nix
+#!nix-shell -I nixpkgs=../../../../../ -i bash -p curl wget yarn2nix

 set -euo pipefail

@@ -11,7 +11,28 @@ fi

 RIOT_WEB_SRC="https://raw.githubusercontent.com/vector-im/element-desktop/$1"

-wget "$RIOT_WEB_SRC/package.json" -O element-desktop-package.json
-wget "$RIOT_WEB_SRC/yarn.lock" -O element-desktop-yarndeps.lock
+# Here we deal with the so-called hakDependencies. They are not part of yarn.lock.
+# Upstream doesn't add them to the dependencies field, because they want to prevent
+# the install scripts to be run by npm/yarn. Fortunately, yarn2nix doesn't run
+# install scripts by default, so it's okay to add them to the dependencies for us.
+# For more information, read the description at
+# https://github.com/vector-im/element-desktop/tree/v1.7.17/scripts/hak
+
+TMPDIR="$(mktemp -d)"
+trap "rm -rf $TMPDIR;" EXIT
+
+pushd "$TMPDIR"
+
+curl "$RIOT_WEB_SRC/package.json" \
+       | jq '. + { dependencies: (.dependencies + .hakDependencies) }' \

[ghost]

It turns out to be much more complicated, because the cargo build of seshat-node-native calls npm install, so we have multiple layers of npm calling yarn calling cargo calling npm. With lots of things being downloaded and executed. It's a huge shitshow.

[teutat3s]

Is there anything we could do to help fix building element-desktop with encrypted chat search working on NixOS?

[ghost]

I poured some (many) more hours into this today. I can build the two native modules, matrix-seshat and keytar, both in a seperate derivation and as part of the element-desktop build. However, I can not build them correctly against electron's node lib, so when starting the compiled thing, it refuses to load the module. Attempting to build against electron gives a mystic error.

$ nix-build https://github.com/petabyteboy/nixpkgs/archive/feature/seshat1.tar.gz -A element-desktop.seshat-node

Or the second approach, with everything in one derivation using the hak tool:

$ nix-build https://github.com/petabyteboy/nixpkgs/archive/feature/seshat2.tar.gz -A element-desktop

   Compiling notify v4.0.15
   Compiling r2d2 v0.8.9
   Compiling hkdf v0.10.0
   Compiling aes-ctr v0.6.0
   Compiling tempfile v3.2.0
   Compiling r2d2_sqlite v0.17.0
   Compiling pbkdf2 v0.6.0
   Compiling futures-macro v0.3.12
   Compiling thiserror-impl v1.0.23
   Compiling failure v0.1.8
   Compiling futures-util v0.3.12
The following warnings were emitted during compilation:

warning: ar: /build/source/seshat-node/native/target/release/build/neon-sys-f7ab2e53f2e370fb/out/native/build/Release/obj.target/neon/src/neon.o: No such file or directory

error: failed to run custom build command for `neon-sys v0.4.0`

Caused by:
  process didn't exit successfully: `/build/source/seshat-node/native/target/release/build/neon-sys-564e91a635bdfcb5/build-script-build` (exit code: 1)
  --- stdout
  Skipping node-gyp installation as part of npm install.

  > @ build-release /build/source/seshat-node/native/target/release/build/neon-sys-f7ab2e53f2e370fb/out/native
  > node-gyp build

  make: Entering directory '/build/source/seshat-node/native/target/release/build/neon-sys-f7ab2e53f2e370fb/out/native/build'
    CXX(target) Release/obj.target/neon/src/neon.o
  make: Leaving directory '/build/source/seshat-node/native/target/release/build/neon-sys-f7ab2e53f2e370fb/out/native/build'
  TARGET = Some("x86_64-unknown-linux-gnu")
  HOST = Some("x86_64-unknown-linux-gnu")
  AR_x86_64-unknown-linux-gnu = None
  AR_x86_64_unknown_linux_gnu = None
  HOST_AR = None
  AR = Some("ar")
  running: "ar" "cq" "/build/source/seshat-node/native/target/release/build/neon-sys-f7ab2e53f2e370fb/out/libneon.a" "/build/source/seshat-node/native/target/release/build/neon-sys-f7ab2e53f2e370fb/out/native/build/Release/obj.target/neon/src/neon.o"
  cargo:warning=ar: /build/source/seshat-node/native/target/release/build/neon-sys-f7ab2e53f2e370fb/out/native/build/Release/obj.target/neon/src/neon.o: No such file or directory
  exit code: 1

  --- stderr
  In file included from /build/source/seshat-node/.electron-gyp/13.0.1/include/node/node.h:67,
                   from ../../../../../../../../node_modules/nan/nan.h:56,
                   from ../src/neon.cc:2:
  /build/source/seshat-node/.electron-gyp/13.0.1/include/node/v8.h:1670:79: warning: 'using ResolveCallback = class v8::MaybeLocal<v8::Module> (*)(class v8::Local<v8::Context>, class v8::Local<v8::String>, class v8::Local<v8::Module>)' is deprecated: Use ResolveModuleCallback [-Wdeprecated-declarations]
   1670 |                                                       ResolveCallback callback);
        |                                                                               ^
  /build/source/seshat-node/.electron-gyp/13.0.1/include/node/v8.h:1652:9: note: declared here
   1652 |   using ResolveCallback V8_DEPRECATE_SOON("Use ResolveModuleCallback") =
        |         ^~~~~~~~~~~~~~~
  /build/source/seshat-node/.electron-gyp/13.0.1/include/node/v8.h:8652:51: warning: 'using HostImportModuleDynamicallyCallback = class v8::MaybeLocal<v8::Promise> (*)(class v8::Local<v8::Context>, class v8::Local<v8::ScriptOrModule>, class v8::Local<v8::String>)' is deprecated: Use HostImportModuleDynamicallyWithImportAssertionsCallback instead [-Wdeprecated-declarations]
   8652 |       HostImportModuleDynamicallyCallback callback);
        |                                                   ^
  /build/source/seshat-node/.electron-gyp/13.0.1/include/node/v8.h:7291:7: note: declared here
   7291 | using HostImportModuleDynamicallyCallback V8_DEPRECATE_SOON(
        |       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  In file included from ../src/neon.cc:2:
  ../../../../../../../../node_modules/nan/nan.h: In function 'void Nan::AsyncQueueWorker(Nan::AsyncWorker*)':
  ../../../../../../../../node_modules/nan/nan.h:2294:7: warning: cast between incompatible function types from 'void (*)(uv_work_t*)' {aka 'void (*)(uv_work_s*)'} to 'uv_after_work_cb' {aka 'void (*)(uv_work_s*, int)'} [-Wcast-function-type]
   2294 |     , reinterpret_cast<uv_after_work_cb>(AsyncExecuteComplete)
        |       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  In file included from ../src/neon.cc:10:
  ../src/neon_task.h: In member function 'void neon::Task::complete()':
  ../src/neon_task.h:62:70: warning: 'v8::Local<v8::Value> node::MakeCallback(v8::Isolate*, v8::Local<v8::Object>, v8::Local<v8::Function>, int, v8::Local<v8::Value>*)' is deprecated: Use MakeCallback(..., async_context) [-Wdeprecated-declarations]
     62 |     node::MakeCallback(isolate_, context->Global(), callback, 2, argv);
        |                                                                      ^
  In file included from ../../../../../../../../node_modules/nan/nan.h:56,
                   from ../src/neon.cc:2:
  /build/source/seshat-node/.electron-gyp/13.0.1/include/node/node.h:192:50: note: declared here
    192 |                 NODE_EXTERN v8::Local<v8::Value> MakeCallback(
        |                                                  ^~~~~~~~~~~~
  /build/source/seshat-node/.electron-gyp/13.0.1/include/node/node.h:108:42: note: in definition of macro 'NODE_DEPRECATED'
    108 |     __attribute__((deprecated(message))) declarator
        |                                          ^~~~~~~~~~
  In file included from ../src/neon.cc:10:
  ../src/neon_task.h: In function 'void neon::queue_task(neon::Task*)':
  ../src/neon_task.h:98:17: warning: cast between incompatible function types from 'void (*)(uv_work_t*)' {aka 'void (*)(uv_work_s*)'} to 'uv_after_work_cb' {aka 'void (*)(uv_work_s*, int)'} [-Wcast-function-type]
     98 |                 (uv_after_work_cb)complete_task);
        |                 ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  ../src/neon.cc: In function 'size_t Neon_ArrayBuffer_Data(void**, v8::Local<v8::ArrayBuffer>)':
  ../src/neon.cc:217:20: error: 'Contents' is not a member of 'v8::ArrayBuffer'
    217 |   v8::ArrayBuffer::Contents contents = buffer->GetContents();
        |                    ^~~~~~~~
  ../src/neon.cc:218:15: error: 'contents' was not declared in this scope
    218 |   *base_out = contents.Data();
        |               ^~~~~~~~
  ../src/neon.cc: In function 'void Neon_Class_SetClassMap(v8::Isolate*, void*, Neon_DropCallback)':
  ../src/neon.cc:329:41: warning: 'void node::AtExit(void (*)(void*), void*)' is deprecated: Use the three-argument variant of AtExit() or AddEnvironmentCleanupHook() [-Wdeprecated-declarations]
    329 |   node::AtExit(cleanup_class_map, holder);
        |                                         ^
  In file included from ../../../../../../../../node_modules/nan/nan.h:56,
                   from ../src/neon.cc:2:
  /build/source/seshat-node/.electron-gyp/13.0.1/include/node/node.h:866:22: note: declared here
    866 |     NODE_EXTERN void AtExit(void (*cb)(void* arg), void* arg = nullptr));
        |                      ^~~~~~
  /build/source/seshat-node/.electron-gyp/13.0.1/include/node/node.h:108:42: note: in definition of macro 'NODE_DEPRECATED'
    108 |     __attribute__((deprecated(message))) declarator
        |                                          ^~~~~~~~~~
  In file included from ../src/neon.cc:8:
  ../src/neon_string.h: In member function 'v8::Local<v8::String> neon::Slice::ToJsString(v8::Isolate*, const char*)':
  ../src/neon_string.h:28:18: warning: ignoring return value of 'bool v8::MaybeLocal<T>::ToLocal(v8::Local<S>*) const [with S = v8::String; T = v8::String]' declared with attribute 'warn_unused_result' [-Wunused-result]
     28 |     maybe.ToLocal(&result);
        |     ~~~~~~~~~~~~~^~~~~~~~~
  make: *** [neon.target.mk:113: Release/obj.target/neon/src/neon.o] Error 1
  gyp ERR! build error
  gyp ERR! stack Error: `make` failed with exit code: 2
  gyp ERR! stack     at ChildProcess.onExit (/nix/store/009815w1n26nl10rgffgahk7aka80p1m-nodejs-14.17.0/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:194:23)
  gyp ERR! stack     at ChildProcess.emit (events.js:376:20)
  gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)
  gyp ERR! System Linux 5.11.21
  gyp ERR! command "/nix/store/009815w1n26nl10rgffgahk7aka80p1m-nodejs-14.17.0/bin/node" "/nix/store/009815w1n26nl10rgffgahk7aka80p1m-nodejs-14.17.0/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "build"
  gyp ERR! cwd /build/source/seshat-node/native/target/release/build/neon-sys-f7ab2e53f2e370fb/out/native
  gyp ERR! node -v v14.17.0
  gyp ERR! node-gyp -v v5.1.0
  gyp ERR! not ok
  npm ERR! code ELIFECYCLE
  npm ERR! errno 1
  npm ERR! @ build-release: `node-gyp build`
  npm ERR! Exit status 1
  npm ERR!
  npm ERR! Failed at the @ build-release script.
  npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

  npm ERR! A complete log of this run can be found in:
  npm ERR!     /tmp/.npm/_logs/2021-05-29T21_41_18_532Z-debug.log


  error occurred: Command "ar" "cq" "/build/source/seshat-node/native/target/release/build/neon-sys-f7ab2e53f2e370fb/out/libneon.a" "/build/source/seshat-node/native/target/release/build/neon-sys-f7ab2e53f2e370fb/out/native/build/Release/obj.target/neon/src/neon.o" with args "ar" did not execute successfully (status code exit code: 1).


warning: build failed, waiting for other jobs to finish...
The following warnings were emitted during compilation:

warning: ar: /build/source/seshat-node/native/target/release/build/neon-sys-400bc8aadcaf0046/out/native/build/Release/obj.target/neon/src/neon.o: No such file or directory

error: build failed
neon ERR! cargo build failed

Error: cargo build failed
    at Target.<anonymous> (/build/source/seshat-node/node_modules/neon-cli/lib/target.js:121:35)
    at step (/build/source/seshat-node/node_modules/neon-cli/lib/target.js:32:23)
    at Object.next (/build/source/seshat-node/node_modules/neon-cli/lib/target.js:13:53)
    at fulfilled (/build/source/seshat-node/node_modules/neon-cli/lib/target.js:4:58)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
builder for '/nix/store/vs8qn1fzq0cbh5830pci0mxr8fiqdx89-seshat-node-2.2.4.drv' failed with exit code 1
error: build of '/nix/store/vs8qn1fzq0cbh5830pci0mxr8fiqdx89-seshat-node-2.2.4.drv' on 'ssh://pbb@mozarella.petabyte.dev' failed: builder for '/nix/store/vs8qn1fzq0cbh5830pci0mxr8fiqdx89-seshat-node-2.2.4.drv' failed with exit code 1
builder for '/nix/store/vs8qn1fzq0cbh5830pci0mxr8fiqdx89-seshat-node-2.2.4.drv' failed with exit code 1
error: build of '/nix/store/vs8qn1fzq0cbh5830pci0mxr8fiqdx89-seshat-node-2.2.4.drv' failed

<rant>
Honestly it's just a horrible build system. hak and neon and so on don't give any consideration to users who want to build the libraries in an environment without an internet connection.
</rant>

Is there anything we could do to help fix building element-desktop with encrypted chat search working on NixOS?

@teutat3s If you can find a fix for the error, there shouldn't be much work left.

[ghost]

I think I found the solution to that error: Upstream is using Electron 12.x while we are using Electron 13.x.

[ghost]

PR incoming ✨