Missing pw_cost key error after I updated encryption version to 004

[rdhyee]

After I updated the encryption version from 004 from 003 (https://standardnotes.org/help/security), I started to encounter the following exception when running standardnotes-fs:

Traceback (most recent call last):
  File "/Users/raymondyee/.pyenv/versions/myenv/bin/snfs", line 8, in <module>
    sys.exit(main())
  File "/Users/raymondyee/C/src/my_conda_envs/myenv-3.8.5/.venv/lib/python3.8/site-packages/standardnotes_fs/standardnotes_fs.py", line 205, in main
    keys = sn_api.gen_keys(password)
  File "/Users/raymondyee/C/src/my_conda_envs/myenv-3.8.5/.venv/lib/python3.8/site-packages/standardnotes_fs/api.py", line 73, in gen_keys
    pw_cost = pw_info['pw_cost']
KeyError: 'pw_cost'

Has anyone else run into this problem? I've been running standardnotes-fs with few problems for over a year before today. I'll try to debug and fix this issue.

[tannercollin]

Unfortunately version 004 isn't supported by snfs :(

This goes for all the alternative SN clients. I don't think it's likely any of them will be updated to support 004 because of the amount of work involved.

[rdhyee]

@tannercollin Thanks for letting me know that it would a lot of work to get support in place for 004. I'm going to roll back to 003 because snfs is such an important part of my workflow!

[rdhyee]

@tannercollin Thank you so much for snfs. I'm going to close this issue because for the time being it sounds like the best thing I can do right now is roll back to 003 and live with that.

[tannercollin]

You're welcome!

I just fixed a bug and also added an error message if 004 is detected. Please upgrade snfs.

[tannercollin]

@mobitar is there any way to downgrade an account to 003 in case a person doesn't have backups?

[moughxyz]

You can (at your own risk) perform the following steps:

1. Download a decrypted backup of your data
2. Delete your account
3. Download a pre-3.5 client from our desktop repo Releases page
4. Register for an account
5. Import your data

Your new account will use 003.

[rdhyee]

@tannercollin Thanks again for snfs and for continuing to fix bugs. I've updated to the new code but I still can't get snfs to work with my recovered setup. I'm going to try again with the advice from @mobitar -- I need to try a pre-3.5 desktop release.

[rdhyee]

Partial success. I logged off of SN everywhere, deleted my account, set up a new account using SN desktop on my mac (version 3.4.10). snfs works again for me. (big yay). I restored my backup that uses 003. app-legacy.standardnotes.org also works.

What doesn’t work fully:

1. app.standardnotes.org the notes seem to be decrypted properly, but the tags are all marked with “missing keys”
2. The apps on Android and the iPad. I run pretty much the latest versions of all the apps (and I’m a beta user). On my Pixel 2 (3 years old) — the app dies in the process of trying to decrypt my 1700+ notes. On the iPad (3 yeears old), it’s a mess — all I see is “Missing keys” 294/294 notes and tags encrypted — I have > 1700 notes.

[rdhyee]

Now, I'm getting closed to getting things working like they were. More success: I updated my mac SN client to 3.5.14 and that enabled me to get my Android SN client to be happy again. Also app.standardnotes.org started working for me. Presumably the mac app will start working for me soon….

[tannercollin]

The newest mobile app versions are fine working with 003? They don't try to upgrade the encryption?

[rdhyee]

@tannercollin The newest apps work fine for me after I used the 3.4.10 macos desktop app to load up my 003 backups and then upgrade my desktop app to 3.5.15 to get some magic cleaned up version to the server, which the latest Android and iPadOS apps could then be happy with. Does that make sense? Getting the sequencing of restoring my set up was the painful part here.

[rdhyee]

@mobitar @tannercollin Well, that was an adventure. I had grown so dependent on running Standard Notes on my two laptops, my Android phone, my iPad + snfs on my laptops that it was disruptive to have had part of that workflow fall apart when I thoughtlessly upgrade from 003 to 004. (I'm generally conservative about upgrading precisely because I want to avoid untimely disruptions like this.)

@mobitar It seems that in the longer time frame, if people want to keep running snfs, we'll have to implement 004. What pointers can you supply me so that I can see what I can to do that implementation (with guidance from @tannercollin if he has time.) I'm motivated to do an implementation if it's not way too hard and if there's enough time to do it before the plug on 003 is pulled.

[tannercollin]

I'd love it if you implemented 004 and I can provide guidance. There used to be a spec at this link, but it must have been moved:

https://github.com/standardnotes/snjs/blob/004/specification.md

Edit, found it here: https://github.com/standardnotes/snjs/blob/master/specification.md

Also note that we have the luxury of only implementing the bare minimum to get it to work. For example, we don't have to implement 003 -> 004 migration because we can tell the user to do it in an official client, etc.

[moughxyz]

My number one advice is to find a way to get JavaScript running in your env ;) This way you can take advantage of using SNJS. But probably not realistic in this case. Otherwise, there's really no way around it: you have to read the specification, and build the equivalent in Python (while also using our implementation as a guide).

[pkulak]

Wow, Standard Notes really screwed us here. I hope this new encryption is absolutely amazing and fixes all kinds of security issues, because it just totally hosed the workflow of lots of people. I'll switch to a new system before I backup/delete account/restore new account/whatever.

Does anyone have any ideas for a way to share secrets between a local filesystem/web/mobile that won't be arbitrarily hosed at random? Might have to switch to something really heavy and obnoxious like One Password. Arg.

[rdhyee]

@pkulak As I wrote on the SN tools channel (https://standardnotes.slack.com/archives/CP3U7DY2D/p1611253197002500):

Anyone else out there enjoying the "Note to Self" feature in Signal (https://support.signal.org/hc/en-us/articles/360043272451-Note-to-Self) -- especially combined with the ability to delete messages (https://support.signal.org/hc/en-us/articles/360007320491-Delete-messages-and-alerts)? I used this a lot to send a private key around when configuring Standard Notes on my various devices.

This use of Signal is to share the occasional secret between devices for me. I personally also use 1password.

[tannercollin]

Does Signal note-to-self support local filesystem?

You could also self-host Bitwarden. It has a command line client that you could write a script around.

@rdhyee are you still interested in helping me implement 004? Do you have Telegram Messenger?

[rdhyee]

@tannercollin I'm still interested in doing the implementation but haven't had time to dig in yet. I do have Telegram.

[rdhyee]

I just started to wade into this issue a bit last night. I figure that I should first teach myself the ins and outs of how Encryption Protocol Specification v003 | Standard Notes Documentation works before studying Encryption Protocol Specification v004 | Standard Notes Documentation. And I can convince myself how to use the pieces in the current snfs to decrypt a static download of my notes before then setting up a new test Standard Notes account (using the 004 encryption version)) and figuring out how to decrypt and then encrypt notes using Python.

In other words, I'm just warming up to tackle this implementationl

[johackim]

Awesome, thanks @rdhyee !

[rdhyee]

As I reported on slack (https://standardnotes.slack.com/archives/C3KQ7V2H2/p1612370087018400?thread_ts=1612288313.016100&cid=C3KQ7V2H2):

I installed snjs yesterday but couldn’t figure out how to use it — my JS skills are primitive, and I figure that I need to learn how to use JS modules and try again.

I also haven’t been able to write the Python version of decrypt.html.

What would be really helpful is worked-out, simplest examples of encryption and decryption for 003 and 004 using specific values of uip, pw_cost, pw_nonce so that I can verify that I get the easiest case working.

I plan to write out more fully what I’ve attempted so far and where I’ve gotten stuck.

[tannercollin]

I've implemented a protocol v004 decryption demo in Python 3 here:

https://gist.github.com/tannercollin/d7474c6afba0dc2026dd996b9dedf197

Save it as a .py file and install the requirements listed. You can go through it step by step to see how the crypto works.

[rdhyee]

I've not forgotten about this issue. I'm getting some experience working with the sync API on 003 and then will work on integrating 004 into the codebase -- unless someone beats me to it.

[johackim]

Hello @rdhyee @tannercollin 😀

Any news about this issue ?

I can't help technically but I can help with a bounty if needed.

Thanks !

[rdhyee]

@johackim @tannercollin I've not forgotten about this issue but have been struggling to find time to work on it. I hope to get some time in the next week or two.

[rdhyee]

@tannercollin OK, a first very rough cut at incorporating your demo code for v004 into snfs: master...rdhyee:v004

I was able to run

import logging
import standardnotes_fs
from standardnotes_fs import api
from standardnotes_fs.api import SNAPIException, StandardNotesAPI
from standardnotes_fs.itemmanager import ItemManager
from standardnotes_fs.crypt import EncryptionHelper


username = "[USERNAME]"
password = "[PASSWORD]"

sync_url = "https://sync.standardnotes.org"
keys = None


try:
    sn_api = StandardNotesAPI(sync_url, username)
    if not keys:
        keys = sn_api.gen_keys(password)
        del password
    keys = sn_api.sign_in(keys)
    log_msg = 'Successfully logged into account "%s".'
    logging.info(log_msg % username)
    login_success = True
except SNAPIException as e:
    print(e)
except ConnectionError:
    log_msg = 'Unable to connect to the sync server at "%s".'
    print(log_msg % sync_url)
    sys.exit(1)
except MissingSchema:
    log_msg = 'Invalid sync server url "%s".'
    print(log_msg % sync_url)
    sys.exit(1)

ext = ".md"
item_manager = ItemManager(sn_api, ext)

item_manager.sync_items()
print(item_manager.get_notes())

[krillin666]

@tannercollin OK, a first very rough cut at incorporating your demo code for v004 into snfs: master...rdhyee:v004

I was able to run

import logging
import standardnotes_fs
from standardnotes_fs import api
from standardnotes_fs.api import SNAPIException, StandardNotesAPI
from standardnotes_fs.itemmanager import ItemManager
from standardnotes_fs.crypt import EncryptionHelper


username = "[USERNAME]"
password = "[PASSWORD]"

sync_url = "https://sync.standardnotes.org"
keys = None


try:
    sn_api = StandardNotesAPI(sync_url, username)
    if not keys:
        keys = sn_api.gen_keys(password)
        del password
    keys = sn_api.sign_in(keys)
    log_msg = 'Successfully logged into account "%s".'
    logging.info(log_msg % username)
    login_success = True
except SNAPIException as e:
    print(e)
except ConnectionError:
    log_msg = 'Unable to connect to the sync server at "%s".'
    print(log_msg % sync_url)
    sys.exit(1)
except MissingSchema:
    log_msg = 'Invalid sync server url "%s".'
    print(log_msg % sync_url)
    sys.exit(1)

ext = ".md"
item_manager = ItemManager(sn_api, ext)

item_manager.sync_items()
print(item_manager.get_notes())

Nice work, keep going !

[tannercollin]

Looks great so far, I've pulled it into a v004 branch in the main repo.

[johackim]

Hello @tannercollin and @rdhyee ;)

I tested the v004 branch and I have an error :

Processing item xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    content_type Note
    Decrypting enc_item_key
Traceback (most recent call last):
  File "/home/johackim/Downloads/standardnotes-fs/env/bin/snfs", line 8, in <module>
    sys.exit(main())
  File "/home/johackim/Downloads/standardnotes-fs/env/lib/python3.9/site-packages/standardnotes_fs/standardnotes_fs.py", line 260, in main
    fuse = FUSE(StandardNotesFUSE(sn_api, sync_sec, args.ext),
  File "/home/johackim/Downloads/standardnotes-fs/env/lib/python3.9/site-packages/standardnotes_fs/sn_fuse.py", line 26, in __init__
    self.item_manager = ItemManager(sn_api, ext)
  File "/home/johackim/Downloads/standardnotes-fs/env/lib/python3.9/site-packages/standardnotes_fs/itemmanager.py", line 272, in __init__
    self.sync_items()
  File "/home/johackim/Downloads/standardnotes-fs/env/lib/python3.9/site-packages/standardnotes_fs/itemmanager.py", line 96, in sync_items
    response = self.sn_api.sync(dirty_items)
  File "/home/johackim/Downloads/standardnotes-fs/env/lib/python3.9/site-packages/standardnotes_fs/api.py", line 159, in sync
    return self.handle_response_items_004(response)
  File "/home/johackim/Downloads/standardnotes-fs/env/lib/python3.9/site-packages/standardnotes_fs/api.py", line 215, in handle_response_items_004
    version, nonce, ciphertext, encoded_authenticated_data = enc_item_key.split(":")
ValueError: too many values to unpack (expected 4)

Thanks for your time !

[krillin666]

Any update on this ?
This is a great tool and too unfortunate to not be able to use in SN..

Thank you !

[rdhyee]

@krillin666 thanks for your patience -- I've not had time to work on this issue. But fixing snfs for 003 (#35) has gotten me back into this code base -- so I hope to keep going and finish up this work. No promises though....

[johackim]

Any news ? @rdhyee

[rdhyee]

@johackim I've not made any progress on this issue because snfs is broken right now for 003 (because of API changes) and I've not yet been able to fix the problem for 003. I've not given up yet but haven't found time to fix the problem given how busy I've been.

[johackim]

Hello all, any news about this ? 😀

[rdhyee]

@johackim Sorry, no progress to report. I've been so busy with my other responsibilities that I've not found time to work on this issue. Between not being able to run snfs and not being able to run something else of importance to me (bordaigorl/sublime-evernote#219 (comment)), I'm in the middle of migrating my note-taking from Evernote and Standard Notes to Obsidian.md -- where having a collection of interlinked markdown files in a file system is the centerpiece (and not tacked on).

[johackim]

It's exactly why I need this fix, I try to migrate my notes from Standard Notes to Obsidian too and I don't know how to do it quickly. Any idea ?

[rdhyee]

@johackim The approach I'm exploring right now -- use Standard Notes to download a decrypted backup of all your notes -- see https://standardnotes.com/help/14/how-do-i-create-and-import-backups-of-my-standard-notes-data. The menu flow I used was

Account -> Data Backups -> Download Backup (Decrypted)

You get a zip file that you can then unzip.

I'm going then look into writing Python scripts to ingest the unzipped files into a format to import into Obsidian.

Hope that helps.