Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cross-signing [2/4] - upload/download keys #5769

Merged

Conversation

@uhoreg
Copy link
Member

uhoreg commented Jul 25, 2019

This is just #4970, with #5759 split off.

Created a new PR so that the review comments don't disappear, since this would be a force-push.

uhoreg added 2 commits Jul 25, 2019
This is a prerequisite for cross-signing, as it allows us to create other things
that live within the device namespace, so they can be used for signatures.
@uhoreg uhoreg requested a review from matrix-org/synapse-core Jul 25, 2019
@richvdh richvdh referenced this pull request Jul 30, 2019
synapse/storage/end_to_end_keys.py Outdated Show resolved Hide resolved
synapse/types.py Outdated Show resolved Hide resolved
@uhoreg

This comment has been minimized.

Copy link
Member Author

uhoreg commented Aug 2, 2019

There seems to be a race condition in the unit test, but I'm not sure how to fix it.

@uhoreg uhoreg requested a review from matrix-org/synapse-core Aug 2, 2019
Copy link
Member

richvdh left a comment

generally looking sane, a few nits

synapse/storage/end_to_end_keys.py Outdated Show resolved Hide resolved
synapse/storage/end_to_end_keys.py Outdated Show resolved Hide resolved
synapse/handlers/e2e_keys.py Outdated Show resolved Hide resolved
synapse/handlers/e2e_keys.py Outdated Show resolved Hide resolved
synapse/handlers/e2e_keys.py Outdated Show resolved Hide resolved
synapse/handlers/e2e_keys.py Outdated Show resolved Hide resolved
synapse/handlers/e2e_keys.py Outdated Show resolved Hide resolved
@@ -145,3 +147,66 @@ def test_claim_one_time_key(self):
"one_time_keys": {local_user: {device_id: {"alg1:k1": "key1"}}},
},
)

@defer.inlineCallbacks
def test_replace_master_key(self):

This comment has been minimized.

Copy link
@richvdh

richvdh Aug 13, 2019

Member

as a general note for future reference, we're moving away from this style of test towards ones which (a) inherit from HomeserverTestCase to instantiate a Homeserver with a mock reactor and (b) exercise the REST API rather than the mid-layer of the handlers.

@richvdh

This comment has been minimized.

Copy link
Member

richvdh commented Aug 13, 2019

There seems to be a race condition in the unit test, but I'm not sure how to fix it.

Yeah, odd. Can you reproduce it locally? If so, and if you turn the verbosity right up (SYNAPSE_TEST_LOG_LEVEL=DEBUG), do the logs (particularly the SQL queries) enlighten you?

@uhoreg

This comment has been minimized.

Copy link
Member Author

uhoreg commented Aug 21, 2019

OK, I think I know what's up with the unit test. When I fetch the cross-signing key, I order by timestamp and pick the first one. But the timestamp only has 1-second resolution, and the unit tests run too fast, so the two keys get stored with the timestamp. So when it fetches the keys, it basically picks one at random. (I guess the fact that it's never happened on my laptop means that my laptop is slow enough that the keys don't have the same timestamp. Or it's just extremely lucky.)

So, what's the best way of ensuring that the key that was last stored is the one that gets selected? Normally, I would use an autoincrement, but it seems that the synapse db doesn't have any autoincrement fields.

@codecov

This comment has been minimized.

Copy link

codecov bot commented Aug 21, 2019

Codecov Report

❗️ No coverage uploaded for pull request base (uhoreg/e2e_cross-signing_merged@72d296a). Click here to learn what that means.
The diff coverage is 49.38%.

@@                        Coverage Diff                         @@
##             uhoreg/e2e_cross-signing_merged    #5769   +/-   ##
==================================================================
  Coverage                                   ?   63.45%           
==================================================================
  Files                                      ?      331           
  Lines                                      ?    36668           
  Branches                                   ?     6068           
==================================================================
  Hits                                       ?    23267           
  Misses                                     ?    11729           
  Partials                                   ?     1672
uhoreg added 3 commits Aug 21, 2019
@uhoreg uhoreg changed the base branch from uhoreg/e2e_cross-signing2-part0 to uhoreg/e2e_cross-signing_merged Aug 29, 2019
@uhoreg

This comment has been minimized.

Copy link
Member Author

uhoreg commented Aug 29, 2019

Switching to using stream ID generator seems to have fixed the tests.

The sytest failure I think is not-my-fault, since it's only failing in the worker sytest, and the sytest that's running against it isn't even calling any cross-signing functionality, but it's hard to tell because it doesn't seem to be copying the sytest artifacts properly.

hawkowl and others added 9 commits Aug 30, 2019
This reverts commit 71fc040.

This broke 3PID invites as #5892 was required for it to work correctly.
Adds a flag to `/versions`' `unstable_features` section indicating that this Synapse understands what an `id_access_token` is, as per #5927 (comment)

Fixes #5927
Python will return a tuple whether there are parentheses around the returned values or not.

I'm just sick of my editor complaining about this all over the place :)
This reverts commit 4765f0c.
Signed-off-by: Aaron Raimist <aaron@raim.ist>
* fix thumbnail storage location

Signed-off-by: Lorenz Steinert <lorenz@steinerts.de>

* Add changelog file.

Signed-off-by: Lorenz Steinert <lorenz@steinerts.de>

* Update Changelog

Signed-off-by: Lorenz Steinert <lorenz@steinerts.de>
@richvdh
richvdh approved these changes Sep 2, 2019
Copy link
Member

richvdh left a comment

lgtm now, modulo the build failure. Apparently @hawkowl has updated the build scripts to fix the artifact uploads, so could you merge latest develop in?

@richvdh

This comment has been minimized.

Copy link
Member

richvdh commented Sep 2, 2019

Sounds like you've fixed it, either way, but:

But the timestamp only has 1-second resolution,

1 millisecond, surely?

anoadragon453 and others added 15 commits Sep 2, 2019
These methods were part of the v1 C/S API. Remove them as they are no longer used by any code paths.
Trace device list changes.
Remove all the "double return" statements which were a result of us removing all the instances of

```
defer.returnValue(...)
return
```

statements when we switched to python3 fully.
* Ensure an auth instance is available to ListMediaInRoom

Fixes #5737

* Changelog
…D/GID (#5970)

Adjust su-exec to only be used if needed.

If UID == getuid() and GID == getgid() then we do not need to su-exec, and chmod will not work.
* Ensure the list media admin API is always available

This API is required for some external media repo implementations to operate (mostly for doing quarantine operations on a room).

* changelog
Previously the stats were not being correctly populated.
Very small code cleanup.
Removes the `bind_email` and `bind_msisdn` parameters from the `/register` C/S API endpoint as per [MSC2140: Terms of Service for ISes and IMs](https://github.com/matrix-org/matrix-doc/pull/2140/files#diff-c03a26de5ac40fb532de19cb7fc2aaf7R107).
@uhoreg uhoreg merged commit 19bb5c8 into uhoreg/e2e_cross-signing_merged Sep 4, 2019
20 checks passed
20 checks passed
buildkite/synapse Build #4044 passed (15 minutes, 27 seconds)
Details
buildkite/synapse/check-sample-config Passed (1 minute, 16 seconds)
Details
buildkite/synapse/check-style Passed (1 minute, 16 seconds)
Details
buildkite/synapse/isort Passed (19 seconds)
Details
buildkite/synapse/mypy Passed (1 minute, 41 seconds)
Details
buildkite/synapse/newspaper-newsfile Passed (17 seconds)
Details
buildkite/synapse/packaging Passed (19 seconds)
Details
buildkite/synapse/pipeline Passed (8 seconds)
Details
buildkite/synapse/python-3-dot-5-slash-postgres-9-dot-5 Passed (11 minutes, 43 seconds)
Details
buildkite/synapse/python-3-dot-5-slash-sqlite Passed (9 minutes, 22 seconds)
Details
buildkite/synapse/python-3-dot-5-slash-sqlite-slash-old-deps Passed (13 minutes, 16 seconds)
Details
buildkite/synapse/python-3-dot-6-slash-sqlite Passed (8 minutes, 45 seconds)
Details
buildkite/synapse/python-3-dot-7-slash-postgres-11 Passed (10 minutes, 48 seconds)
Details
buildkite/synapse/python-3-dot-7-slash-postgres-9-dot-5 Passed (11 minutes, 44 seconds)
Details
buildkite/synapse/python-3-dot-7-slash-sqlite Passed (9 minutes, 23 seconds)
Details
buildkite/synapse/sytest-python-3-dot-5-slash-postgres-9-dot-6-slash-monolith Passed (6 minutes, 26 seconds)
Details
buildkite/synapse/sytest-python-3-dot-5-slash-postgres-9-dot-6-slash-workers Passed (7 minutes, 31 seconds)
Details
buildkite/synapse/sytest-python-3-dot-5-slash-sqlite-slash-monolith Passed (5 minutes, 52 seconds)
Details
buildkite/synapse/trigger-webhook Passed (7 seconds)
Details
coverage/coveralls First build on uhoreg/e2e_cross-signing2-part1 at 64.751%
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.