New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migration to Simperium API needed by October 2018 #64

Closed
qcu opened this Issue Aug 15, 2018 · 17 comments

Comments

Projects
3 participants
@qcu

qcu commented Aug 15, 2018

The deprecated API issue (#63) is resolved, for now, so we should be able to sync again temporarily.

@roundhill said:

That said, we will be deprecating the old API in October so a migration to the Simperium API is still needed here.

The API this is currently using hasn't been supported for many years. We can provide an API key that will allow access to Simplenote.

Simperium HTTP - API reference

Simperium Python - Getting Started

@swalladge

This comment has been minimized.

Collaborator

swalladge commented Aug 15, 2018

I'm currently in discussion with Automattic with the aim of obtaining an API key and find out anything else required to begin porting sncli to the new api.

@swalladge

This comment has been minimized.

Collaborator

swalladge commented Aug 20, 2018

Update: Simplenote has provided me with details required to use sncli with the new api. I'll begin working on porting sncli to Simperium within the next couple of weeks (as soon as I get some free time).

@swalladge swalladge added this to To do in v0.3.0 Aug 20, 2018

@swalladge

This comment has been minimized.

Collaborator

swalladge commented Aug 28, 2018

The upstream python library for Simperium (which would be nice to use instead of rolling our own) doesn't currently support python3. Issue reported at Simperium/simperium-python#10 with my thoughts.

@swalladge

This comment has been minimized.

Collaborator

swalladge commented Sep 4, 2018

Update: I have successfully ported the simperium python client to python 3. It can be found at https://github.com/swalladge/simperium-python3

I have done some tinkering with sncli to use the new library - auth is working fine, but getting the notes sync will be interesting due to changes to the note representation from the server. For example, we can no longer rely on syncnum (gone), version is no longer part of the note object, but returned with the note index function, etc.

Help wanted to test the python3 port and with porting sncli to the new api! 😃

swalladge added a commit that referenced this issue Sep 4, 2018

wip port to simperium-python backend
what works:

- auth
- pulling/pushing notes (sort of)

what doesn't work:

- reliable syncing

notes:

- more information is returned from the `index` method, and less in the
  `get` method - `v` (the version) is only part of the `index` call.
- no more `syncnum` and friends for syncing help. I'm guessing we'll need to
  use the `since` param (not a datetime) to `index` to help with partial syncing.
- while the simperium api docs are comprehensive, there are no docs for
  simplenote specific things. For example, the delete endpoint returns a 400
  client error on attempting to delete a note that doesn't have the `deleted` key
  set to true.

todo:

- work out a syncing algorithm that fits with sncli's note model and simperium.
- handle http errors (no connection, bad requests, not found, etc.)
- test everything!

Thoughts, comments, code review, suggestions, etc. all appreciated! I'll need
some help if I'm to complete this by October!

ref tracking issue #64
@swalladge

This comment has been minimized.

Collaborator

swalladge commented Sep 4, 2018

Please see pull request. ^

@nbenm

This comment has been minimized.

nbenm commented Sep 11, 2018

Hi swalladge
I have installed successfuly simperium-python (python2). Everything works fine.
I have tried to installed yours but install fails.
I explain:

  • my default is python2. I dont't want to change this
  • I have used sudo -H pip install git+https://github.com/swalladge/simperium-python3.git to install your python3 simperium (Simperium3)
  • I have the error: Simperium3 requires Python '>=3' but the running Python is 2.7.15
    Maybe this means you use python and not python3 in some scripts.
    Do you have any idea to sole this?
@swalladge

This comment has been minimized.

Collaborator

swalladge commented Sep 11, 2018

@nbenm sncli only supports python3, and therefore requires the python3 port of the simperium python client.

(You may need to use pip3 or python3 in your commands, or use a virtual environment, so that python 3 is used.)

@nbenm

This comment has been minimized.

nbenm commented Sep 11, 2018

@swalladge

This comment has been minimized.

Collaborator

swalladge commented Sep 11, 2018

I don't understand. Did you try installing the simperium3 package either in a python3 virtual environment or with pip3?

@nbenm

This comment has been minimized.

nbenm commented Sep 11, 2018

@nbenm

This comment has been minimized.

nbenm commented Sep 11, 2018

This is my first try.
I have created a new Simplenote account.
I have then created notes from web and from sncli.
Common actions work well.

I have tried to import my 229 old notes to the new account

for file in ../Simplenote/*.json
do
	cat $file|sncli3 -c ~/.snclirc3 import -
	sleep 1
done

Only 166/229 have been imported.
There are many errors in sncli.log.
Here are some:

2018-09-11 15:41:48,719 [DEBUG] Starting new HTTPS connection (1): simple-note.appspot.com:443
2018-09-11 15:41:49,087 [DEBUG] https://simple-note.appspot.com:443 "POST /api/login HTTP/1.1" 500 0
2018-09-11 15:41:49,090 [DEBUG] AUTHENTICATE: login failed with status 500, check credentials
2018-09-11 15:41:49,093 [DEBUG] Starting new HTTPS connection (1): simple-note.appspot.com:443
2018-09-11 15:41:49,474 [DEBUG] https://simple-note.appspot.com:443 "POST /api2/data?email=toto%2540dagami.org HTTP/1.1" 500 0
2018-09-11 15:41:49,477 [DEBUG] RESPONSE ERROR: 500 Server Error: Internal Server Error for url: https://simple-note.appspot.com/api2/data?email=toto%2540dagami.org
2018-09-11 15:41:49,477 [DEBUG] ERROR: Failed to sync note to server (key=eb71fab4612643fe81fce5ff3e7b2959)
2018-09-11 15:41:49,480 [DEBUG] Starting new HTTPS connection (1): simple-note.appspot.com:443
2018-09-11 15:41:49,838 [DEBUG] https://simple-note.appspot.com:443 "POST /api/login HTTP/1.1" 500 0
2018-09-11 15:41:49,842 [DEBUG] AUTHENTICATE: login failed with status 500, check credentials
2018-09-11 15:41:49,846 [DEBUG] Starting new HTTPS connection (1): simple-note.appspot.com:443
2018-09-11 15:41:50,213 [DEBUG] https://simple-note.appspot.com:443 "POST /api2/data?email=toto%2540dagami.org HTTP/1.1" 401 0
2018-09-11 15:41:50,216 [DEBUG] RESPONSE ERROR: 401 Client Error: Unauthorized for url: https://simple-note.appspot.com/api2/data?email=toto%2540dagami.org
2018-09-11 15:41:50,217 [DEBUG] ERROR: Failed to sync note to server (key=ee6149f9dab14d199e5d4c28c10205fd)
2018-09-11 15:41:50,977 [DEBUG] Starting new HTTPS connection (1): simple-note.appspot.com:443
2018-09-11 15:41:51,335 [DEBUG] https://simple-note.appspot.com:443 "POST /api/login HTTP/1.1" 500 0
2018-09-11 15:41:51,338 [DEBUG] AUTHENTICATE: login failed with status 500, check credentials
2018-09-11 15:41:51,340 [DEBUG] Starting new HTTPS connection (1): simple-note.appspot.com:443
2018-09-11 15:41:51,701 [DEBUG] https://simple-note.appspot.com:443 "GET /api2/index?email=toto%2540dagami.org&length=100 HTTP/1.1" 401 0
2018-09-11 15:41:51,705 [DEBUG] ERROR: Failed to get note list from server
2018-09-11 15:41:51,705 [DEBUG] Saved note to disk (key=efa57d50c0ba4c5c8a179d774dfe5886)
2018-09-11 15:41:51,705 [DEBUG] Full sync completed

It looks like sncli make retries, and some seem to work.

By all my old notes haven't been imported.

I'll do more tests

@nbenm

This comment has been minimized.

nbenm commented Sep 11, 2018

Here's an other issue:
If I delete a note via the web, and then sync with sncli, the note is not removed with sncli sync.
Nothing is logged for this action.
But trashing from sncli moves correctly the note to trash

@swalladge

This comment has been minimized.

Collaborator

swalladge commented Sep 11, 2018

@nbenm thanks for testing! Yes, I think I need to write up a new syncing algorithm for sncli - the old one isn't going to work with the new api.

@swalladge

This comment has been minimized.

Collaborator

swalladge commented Sep 12, 2018

@nbenm Oh I just read through the logs you posted. It seems that it is still the old sncli version since it mentions "simple-note.appspot.com:443" which is the old api.

@swalladge

This comment has been minimized.

Collaborator

swalladge commented Sep 12, 2018

I'm working on a new syncing algorithm now. Perhaps refrain from testing the next branch until I've got something somewhat working.

I would also stress that if you want to help test the new parts I'm working on, please use at own risk. Either use an account that doesn't have anything important saved on it, or make a full backup of your notes first. I really appreciate help with testing, but I don't want anyone to lose any notes!

@nbenm

This comment has been minimized.

nbenm commented Sep 12, 2018

@swalladge

This comment has been minimized.

Collaborator

swalladge commented Sep 13, 2018

@nbenm the latest commit I added makes it somewhat usable. Feel free to test. See the latest commit message for known issues. :)

PS: please continue conversation about the code/bugs/testing in the PR thread #67. That will leave this thread clear for general discussion. Also it will be easier to line up code/testing discussion with the progress on the code.

@swalladge swalladge moved this from To do to In progress in v0.3.0 Sep 13, 2018

@swalladge swalladge closed this Sep 23, 2018

v0.3.0 automation moved this from In progress to Done Sep 23, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment