Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issues (on Linux) with KeepassXC upgrade to version 2.7.3 #77

Closed
mdonz opened this issue Oct 25, 2022 · 37 comments
Closed

Issues (on Linux) with KeepassXC upgrade to version 2.7.3 #77

mdonz opened this issue Oct 25, 2022 · 37 comments
Assignees
Labels
bug Something isn't working
Milestone

Comments

@mdonz
Copy link

mdonz commented Oct 25, 2022

Being a happy user of keepassxc-mail for a long time (Ubuntu 22.04, KeepassXC 2.7.1, KeePassXC-Browser 1.8.3, keepassxc-mail 1.3), I experience issue with a recent upgrade for KeepassXC from former version 2.7.1 to the new version 2.7.3.
There is a discussion going on here: keepassxreboot/keepassxc#8634

The described workaround to get it going is to "Delete the old proxy socket from XDG_RUNTIME_DIR". I did so and my browser integrations (Firefox and Chromium) are working again as expected with the new Keepassxc version 2.7.3 but I cannot get keepassxc-mail working.

Have reverted back to Keepassxc 2.7.1 for now again with everything (Firefox, Chromium & Thunderbird with keepassxc-mail working) but wonder what might be need for keepassxc-mail?

Thanks for a great add-on.

@nbrucy
Copy link

nbrucy commented Oct 26, 2022

I've exactly the same issue, with the same configuration.

@kkapsner
Copy link
Owner

Please try keepassxc-mail-1.3.1.zip

@kkapsner kkapsner self-assigned this Oct 26, 2022
@kkapsner kkapsner added the bug Something isn't working label Oct 26, 2022
@kkapsner kkapsner added this to the Version 1.4 milestone Oct 26, 2022
@nbrucy
Copy link

nbrucy commented Oct 26, 2022

Same issue with the updated extension. To go in more in details, some password sare retrieved and it get stuck on others (no popup, TB just is "connecting to the server" forever). If I deactivate and reactivate Keepass-xc Mail (after closing all the popups flashing at me), sometimes everything work.

@marcinp2
Copy link

marcinp2 commented Oct 26, 2022

I confirm the same issue. Version 2.7.3 and KeePassXC-Browser: 1.8.3 doesn't work at all.
The symptoms:

  • connection is closing between them
  • KeePass icon at login pages is unexpected hiding
  • auto filling doesn't work
  • pop-up menu and fill pass feature doesn't work

(tested with FF at Win10)

@kkapsner
Copy link
Owner

@marcinp2: I think you are in the wrong git repository. This is for keepassXC-mail (mail extension for Thunderbird) and not keepassXC-browser

@nbrucy: do you see anything in the console (Ctrl + Shift + J)?

@mdonz
Copy link
Author

mdonz commented Oct 26, 2022

I have upgraded to keepassxc-mail-1.3.1.zip (while using KeepassXC version 2.7.1), then shutdown browser and Thunderbird (so no more active keepassxc-proxy processes after that).

cd $XDG_RUNTIME_DIR
michael@TWT039LT:/run/user/1001$ ls -la
total 8
drwx------ 17 michael michael 580 Oct 26 07:00 .
drwxr-xr-x 3 root root 60 Oct 24 08:34 ..
.. cut..
srwx------ 1 michael michael 0 Oct 26 07:00 org.keepassxc.KeePassXC.BrowserServer
I then uninstalled KeepasXC 2.7.1 and removed manually "org.keepassxc.KeePassXC.BrowserServer".

After upgrade to KeepassXC 2.7.3 and starting it, it looks good (I believe):
michael@TWT039LT:/run/user/1001$ ls -la
total 8
drwx------ 17 michael michael 580 Oct 27 08:56 .
drwxr-xr-x 3 root root 60 Oct 24 08:34 ..
.. cut ..
lrwxrwxrwx 1 michael michael 81 Oct 27 08:56 org.keepassxc.KeePassXC.BrowserServer -> /run/user/1001/app/org.keepassxc.KeePassXC//org.keepassxc.KeePassXC.BrowserServer
Then starting Firefox, the web browser integration is working well so far. Then starting Thunderbird, it sits forever on trying to get access to email account, load messages, etc. Below is the output from the TB console. Hope that helps.

09:00:08.400 1666814408400 addons.xpi WARN Checking /usr/lib/thunderbird/distribution/extensions for addons
09:00:09.987 This page is in Quirks Mode. Page layout may be impacted. For Standards Mode use “”.
hiddenWindow.html
09:00:10.113 1666814410113 addons.webextension.{2ab1b709-ba03-4361-abf9-c50b964ff75d} WARN Loading extension '{2ab1b709-ba03-4361-abf9-c50b964ff75d}': Reading manifest: Warning processing background.scripts: An unexpected property was found in the WebExtension manifest.
09:00:10.909 This page is in Quirks Mode. Page layout may be impacted. For Standards Mode use “”.
46 AddonRepository.jsm:97:12
09:00:11.249 Successfully loaded OpenPGP library librnp.so version 0.16+git20220124.f06439f7.MZLA from /usr/lib/thunderbird/librnp.so RNPLib.jsm:100:15
09:00:11.674 Found 0 public keys and 0 secret keys (0 protected, 0 unprotected) RNPLib.jsm:301:15
09:00:11.839 services.settings: Failed to load last_modified.json: TypeError: NetworkError when attempting to fetch resource. Utils.jsm:330
09:00:11.867 Successfully loaded optional OpenPGP library libgpgme.so.11 from system's standard library locations GPGMELib.jsm:69:13
09:00:11.868 gpgme version: 1.16.0-unknown GPGMELib.jsm:241:15
09:00:12.188 Trying to load /usr/lib/thunderbird/libotr.so OTRLib.jsm:64:11
09:00:12.188 Trying to load libotr.so from system's standard library locations OTRLib.jsm:64:11
09:00:12.188 Trying to load libotr.so.5 from system's standard library locations OTRLib.jsm:64:11
09:00:12.189 Trying to load libotr.so from system's standard library locations OTRLib.jsm:64:11
09:00:12.189 Error: Cannot load required OTR library
loadExternalOTRLib resource:///modules/OTRLib.jsm:109
init resource:///modules/OTRLib.jsm:115
once resource:///modules/OTR.jsm:118
init resource:///modules/OTR.jsm:139
init resource:///modules/OTRUI.jsm:256
nsIPrompt_promptUsernameAndPassword resource:///modules/MsgAsyncPrompter.jsm:553
promptAuth resource:///modules/MsgAsyncPrompter.jsm:479
promptAuth resource:///modules/calendar/utils/calAuthUtils.jsm:236
onPromptStart resource:///modules/calendar/utils/calAuthUtils.jsm:267
onPromptStartAsync resource:///modules/calendar/utils/calAuthUtils.jsm:263
_promiseAuthPrompt resource:///modules/MsgAsyncPrompter.jsm:54
_promiseAuthPrompt resource:///modules/MsgAsyncPrompter.jsm:52
run resource:///modules/MsgAsyncPrompter.jsm:77
OTR.jsm:127:15
09:00:12.214 tb.account.size_on_disk - Trying to set an unsigned scalar to a negative number. 2
09:00:12.361 No chrome package registered for chrome://sendtocategory/content/category_tools.js 2
09:00:12.497 This page is in Quirks Mode. Page layout may be impacted. For Standards Mode use “”.
background.html
09:00:12.692 WebExtensions: KeePassXC-Mail: unable to get smtpEnterPasswordPromptTitle from bundle compose 2 implementation.js:34
09:00:12.693 WebExtensions: KeePassXC-Mail: unable to get imapEnterPasswordPromptTitle from bundle imap implementation.js:34
09:00:12.693 WebExtensions: KeePassXC-Mail: unable to get pop3EnterPasswordPromptTitle from bundle local 2 implementation.js:34
09:00:12.694 WebExtensions: KeePassXC-Mail: unable to get loginDialog.label from bundle wcap 2 implementation.js:34
09:00:12.694 WebExtensions: KeePassXC-Mail: unable to get PromptUsernameAndPassword2 from bundle commonDialog 2 implementation.js:34
09:00:12.695 WebExtensions: KeePassXC-Mail: unable to get PromptPassword2 from bundle commonDialog implementation.js:34
09:00:12.695 WebExtensions: KeePassXC-Mail: unable to get CertPassPromptDefault from bundle pipnss implementation.js:34
09:00:13.778
JSON.parse: unexpected non-whitespace character after JSON data at line 1 column 1520 of the JSON data 2 subprocess_common.jsm:495

@mdonz
Copy link
Author

mdonz commented Oct 26, 2022

I have now converted KeepassXC back to version 2.7.1 (left keepassxc-mail on version 1.3.1).

Everything seems to be working in Thunderbird but now having a look at the TB console, this looks almost identical to what I uploaded earlier. At that time I though that the lines of "WebExtensions: KeePassXC-Mail: unable to get xxxxEnterPasswordPromptTitle" where indicating the issues - but these lines still appear again when TB seems to be working,

The only real difference might be this line (not appearing under a working situation):
JSON.parse: unexpected non-whitespace character after JSON data at line 1 column 1520 of the JSON data 2 subprocess_common.jsm

@nbrucy
Copy link

nbrucy commented Oct 27, 2022

At startup I see the same informations ("WebExtensions: KeePassXC-Mail: unable to get imapEnter ...") as @mdonz.
When I clicked on a folder to get new emails, it get stuck on "Connected to server.org" and nothing is written in the console.

Now if I disable keepassxc-mail, close all the popups, and enable it again, I get a new bunch of ("unable to get imapEnter ...") but with these lines added after it:

2022-10-27 09:47:41.544: Use saved native application de.kkapsner.keepassxc_mail [main.js:26:11](moz-extension://95477d65-[...]-71c3955f14b1/main.js)
09:47:41,545 KeePassXC-Mail: Connecting to native messaging host de.kkapsner.keepassxc_mail [client.js:317:13](moz-extension://95477d65-[...]-71c3955f14b1/from-keepassxc-browser/client.js)
09:47:41,620 KeePassXC-Mail: Server public key: laIuQjTo6[...]mtdAQ=

Now if I clicked on a folder what happen is

09:50:41,292 2022-10-27 09:50:41.291: got credential request: 
Object { host: "imap://server.org", login: "login", loginChangeable: false, openChoiceDialog: true }
[main.js:26:11](moz-extension://95477d65-0441-4330-8976-71c3955f14b1/main.js)
09:50:41,599 2022-10-27 09:50:41.599: keepassXC provided 1 logins

and I get my messages as expected.

@nbrucy
Copy link

nbrucy commented Oct 27, 2022

Now if I try with KeepassXC 2.7.1, always the WebExtensions: KeePassXC-Mail: unable to get imapEnter ... at startup, no messages about connecting to native messaging but it automatically provides the logins for my email account who need to be fetched at startup, as expected.

@kkapsner
Copy link
Owner

The unable to get messages are normal. They are there for backward compatibility with older Thunderbird versions.

I think I will spin up an Ubuntu VM and try to reproduce the problem. But the unexpected whitespace message sounds not good.

@nbrucy
Copy link

nbrucy commented Oct 27, 2022

I did not notice before I've also have the parsing error. Here more details:

JSON.parse: unexpected non-whitespace character after JSON data at line 1 column 816 of the JSON data [subprocess_common.jsm:495](resource://gre/modules/subprocess/subprocess_common.jsm)
    readJSON resource://gre/modules/subprocess/subprocess_common.jsm:495
    waitForCredentials jar:file:///home/user/.thunderbird/iyntdkte.default/extensions/keepassxc-mail@kkapsner.de.xpi!/experiment/implementation.js:1009
    replacement jar:file:///home/user/.thunderbird/iyntdkte.default/extensions/keepassxc-mail@kkapsner.de.xpi!/experiment/implementation.js:406
    _promiseAuthPrompt resource:///modules/MsgAsyncPrompter.jsm:54
    _promiseAuthPrompt resource:///modules/MsgAsyncPrompter.jsm:52
    run resource:///modules/MsgAsyncPrompter.jsm:77
    InterpretGeneratorResume self-hosted:1422
    AsyncFunctionNext self-hosted:632

Then it says:
[Error ] KeePassXC-Mail - Failed to connect: Unknown error

@kkapsner
Copy link
Owner

That's bad. This is a core function of the native messaging that is provided by Mozilla. And the JSON is provided by KeePassXC. So I cannot change any of the involved programs.

@droidmonkey & @varjolintu: I was able to reproduce the same issue in Firefox 106.0.2 with keepassxc-browser 1.8.3:
grafik

It might be related to all the other problems with 2.7.3. (e.g. keepassxreboot/keepassxc#8634, keepassxreboot/keepassxc-browser#1751)
I found out that the JSON created by the proxy (or KeePassXC - not sure where the JSON is created) and sent to the extension may contain invalid JSON:

{"action":"get-logins","message":"9vb[...]wAZ","nonce":"Agn[..]ZM"}{}

Notice the additional {} at the end.

In Thunderbird I'm able to get this output by executing the following in the console (Ctrl + Shift + J):

var {BaseProcess} = Cu.import("resource://gre/modules/subprocess/subprocess_common.jsm");
BaseProcess.create = function create(options){
  let worker = this.getWorker();
  
  return worker.call("spawn", [options]).then(({ processId, fds, pid }) => {
    console.log("creating BaseProcess");
    var baseProcess = new this(worker, processId, fds, pid);
    BaseProcess.last = baseProcess;
    baseProcess.stdout.readJSON =  function readJSON(length) {
      if (!Number.isInteger(length) || length <= 0) {
        throw new RangeError("Length must be a positive integer");
      }
      
      return this.readString(length).then(string => {
        try {
          return JSON.parse(string);
        } catch (e) {
          console.log("Invalid JSON:", string);
          e.errorCode = SubprocessConstants.ERROR_INVALID_JSON;
          throw e;
        }
      });
    };
    return baseProcess;
  });
}

I was not able to inject this code into Firefox.

Some additional information that might help pin the problem down: it only happens on database entries where you "remembered" the choice of allowing the access. If the "Browser request access" pops up the message is fine.

@droidmonkey
Copy link

Yah we know 1.8.3 extension is flaky, there is a fix coming.

@kkapsner
Copy link
Owner

@droidmonkey: I don't think it's the extension. In keepassxc-mail I'm currently using the files from 1.8.0.
This JSON is not generated by the extension but by the other end of the native messaging - i.e. KeePassXC itself.

@mdonz and @nbrucy: I think the only solution for you is at the moment to go back to 2.7.1

@mdonz
Copy link
Author

mdonz commented Oct 27, 2022

@kkapsner I much appreciate your work and help with this. I will stay with 2.7.1 for now. Have just observed an interesting detail (under 2.7.1) which I will raise separately.

@droidmonkey
Copy link

There is a separate problem with 2.7.3 for some Linux distributions where the socket file is not created correctly in the runtime directory. See here for a workaround: keepassxreboot/keepassxc#8634 (comment)

@kkapsner
Copy link
Owner

I started with a complete new VM and never installed 2.7.1 there.
In this folder there is only the snap.keepassXC:
grafik

@kkapsner
Copy link
Owner

The initial connection is working - it just breaks after the invalid JSON:
grafik

I'm also able to reproduce the problem with keepassxc-browser 1.8.0:
grafik

@kkapsner
Copy link
Owner

Just checked: not related to snap - problem also present with PPA version

@kkapsner
Copy link
Owner

@droidmonkey: is there a way to tell KeePassXC to output the native messaging communication? Or do I have to implement a proxy for that?

@mdonz
Copy link
Author

mdonz commented Oct 27, 2022

FYI all of my installed packages are from PPA. No snap involved here.

@kkapsner
Copy link
Owner

It's also broken in my Windows VM.

@droidmonkey
Copy link

You have to run keepassxc in a debugger and break on the message send function to see exactly what we are sending. I am fairly confident we never send a blank json message ({}). You can also use this Python library to dynamically interact with the browser service: https://github.com/hrehfeld/python-keepassxc-browser

@kkapsner
Copy link
Owner

Thanks for the info - will toy around with that.
In the meantime I played with a Windows tool to intercept native messaging. There seems to be blank JSON messages:
grafik
But the message lengths seem correct - so there is also a problem with the code in Thunderbird.

@droidmonkey
Copy link

Cool tool, where can it be found?

@kkapsner
Copy link
Owner

kkapsner commented Oct 27, 2022

https://textslashplain.com/2022/01/08/debug-native-messaging/
No guarantee it's not malicious... I just run it in a VM.

EDIT: Here the git: https://github.com/ericlaw1979/NativeMessagingDebugger/releases

@kkapsner
Copy link
Owner

@droidmonkey: KeePassXC is sending wrong data:
grafik
No idea what causes the problem. But I have to try several get-logins to get the problem. I guess it's some sort of racing condition.

@kkapsner
Copy link
Owner

I actually would be able to to mess with Thunderbird code to fix these messages. But this will not be part of an official release (I gues I would make an additional extension for that).

And it would still leave Firefox broken.

So I would rather prefer KeePassXC to be fixed. @droidmonkey: is there anything more that I can provide you to find the culprit?

@kkapsner
Copy link
Owner

@ericlaw1979: thanks for the great tool

@varjolintu
Copy link

If something sends blank JSON messages, it's KeePassXC 2.7.3. Everything would work normally with 2.7.1.

@droidmonkey
Copy link

Is that part of the async dialog problem?

@varjolintu
Copy link

@droidmonkey Yes. It's related to that.

@mdonz
Copy link
Author

mdonz commented Oct 30, 2022

Good news: (Ubuntu 22.04) - I just applied the update to KeepassXC 2.7.4 and so far, my browser integration is working for Firefox and keepassxc-mail under Thunderbird!
Hoping this will continue. Thanks to all involved!

@kkapsner
Copy link
Owner

I do not see any empty messages with 2.7.4. So I would close this issue.

One last question @droidmonkey and @varjolintu: the empty messages itself were not the problem but that two messages were concatenated and in the protocol the length of the message was therefore wrong (see my last screenshot). Was this also introduced by the asynchronous dialog? Or could that happen again (e.g when the browser and Thunderbird are talking to keepassXC at the exact same time or something similar)?

@droidmonkey
Copy link

It was likely a double tap of json messages, a coding error. We sent the real message and an empty one. I thought the native message debugger showed the empty message was sent with the correct length though.

@varjolintu
Copy link

It is exactly how @droidmonkey described it. The real answer was sent directly to the same socket as the empty message, which caused this error.

@kkapsner
Copy link
Owner

It was likely a double tap of json messages, a coding error. We sent the real message and an empty one. I thought the native message debugger showed the empty message was sent with the correct length though.

Most of the time it showed the correct length. In these cases Thunderbird handled them correctly. But sometimes the length was not correct (in the screenshot above the real message is 431 bytes long + the empty message = 433 => breaking the native messaging connection)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

6 participants