Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(8/8) Clean up integration tests now that Entrys are mockable #3584

Merged
merged 34 commits into from Nov 18, 2019

Conversation

@julianknutsen
Copy link
Contributor

julianknutsen commented Nov 9, 2019

New commits start at 2852148

See wall of text on #3583 for context

julianknutsen added 28 commits Nov 3, 2019
Fix a bug where remove() was called in the addMailboxData()
failure path.

1. Sender's can't remove mailbox entries. Only
   the receiver can remove it so even if the previous add() failed and
   left partial state, the remove() can never succeed.

2. Even if the sender could remove, this path used remove() instead
   of removeMailboxData() so it wouldn't have succeed anyway.

This patch cleans up the failure path as well as adds a precondition
for the remove() function to ensure future callers don't use them for
ProtectedMailboxStorageEntrys.
Add comments and refactor the body in order to make it easier to
understand.
Refactor addProtectedStorageEntry for more readability and add comments
to help future readers.
Refactor for readability and add comments for future readers.
Refactor for readability and add comments to help future readers.
Refactor for readability and add comments for future readers.
Removed duplicate log messages that are handled inside the various helper methods
and print more verbose state useful for debugging.

Updated potentially misleading comments around hashing collisions
…icates

Now returns false on duplicate sequence numbers. This matches more of
the expected behavior for an add() function when the element previously exists.

The only callers are either P2PService users that always increment the
sequence number or the onMessage() handler which doesn't verify the return
so there will be no visible change other than the increased readability
of the code and deduplication of the code paths.
Now returns false if the sequence number of the refresh matches
the last operation seen for the specified hash. This is a more expected
return value when no state change occurs.

The only callers are either P2PService users that always increment the
sequence number or the onMessage() handler which doesn't verify the return
so there will be no visible change other than the increased readability
of the code and deduplication of the code paths.
…duplicate sequence #s

Remove operations are now only processed if the sequence number
is greater than the last operation seen for a specific payload.

The only creator of new remove entrys is the P2PService layer that always increments
the sequence number. So, this is either left around from a time where
removes needed to work with non-incrementing sequence numbers or just
a longstanding bug.

With the completion of this patch, all operations now require increasing
sequence numbers so it should be easier to reason about the behavior in
future debugging.
Use the DI Clock object already available in P2PDataStore, instead
of calling System.currentTimeMillis() directly. These two functions
have the same behavior and switching over allows finer control
of time in the tests.
Deduplicate some code in the ProtectedStorageEntry constructors in
preparation for passing in a Clock parameter.
Switch from System.currentTimeMills() to
Clock.millis() so dependency injection can
be used for tests that need finer control of time.

This involves attaching a Clock to the resolver
so all fromProto methods have one available when they
reconstruct a message. This uses the Injector for the APP
and a default Clock.systemDefaultZone is used in the manual
instantiations.

Work was already done in #3037 to make this possible.

All tests still use the default system clock for now.
Reduces non-deterministic failures of the refreshTTL tests that resulted
from the uncontrollable System.currentTimeMillis().

Now, all tests have extremely fine control over the elapsed time between
calls which makes the current and future tests much better.
Add tests for removing expired entries and optionally purging
the sequence number map. Now possible since these tests have
control over time with the ClockFake.

The remove validation needed to be improved since deletes through
the expire path don't signal HashMap listeners or write sequence numbers.
The original test would take over 5 seconds. Allow tests to set the number
of required entries before purge to a lower value so the tests
can run faster with the same confidence.
The custom code to verify the refreshTTLMessage's signature and update
an entry isn't necessary. Just have the code construct an updated
ProtectedStorageEntry from the existing and new data, verify it,
and add it to the map.

This also allows the removal of the ProtectedStorageEntry APIs
that modify internal state.
The code around validating MailboxStoragePayloads is subtle when
a MailboxStoragePayload is wrapped in a ProtectedStorageEntry. Add tests
to document the current behavior.
Method bodies are copied from P2PDataStore to separate refactoring
efforts and behavior changes.

Identified a bug where a ProtectedMailboxStorageEntry mailbox entry
could be added, but never removed.
Extract code from P2PDataStore, test it, and use new methods.
Now that the objects can answer questions about valid conditions
for add/remove, ask them directly.

This also pushes the logging down into the ProtectedStorageEntry and
ProtectedMailboxStorageEntry and cleans up the message.
Add toString() for ProtectedStorageEntry so log messages have useful
information and clean up the formatting.
Move the signature checks into the objects to clean up the calling code
and make it more testable.

The testing now has to take real hashes so some work was done in the fixtures
to create valid hashable objects.
Move the signature checks into the objects to clean up the calling code
and make it more testable.
This mailbox-only check can now exist inside the object for which it
belongs. This makes it easier to test and moves closer to allowing
the deduplication of the remove() methods.
The current check verifies that the stored Payload.ownerPubKey == stored Entry.ownerPubKey.

This is the same check that was done when the item was originally added
and there is no reason to do it again.
Let the objects compare their metadata instead of doing it for them. This
allows for actual unit testing and paves the way for deduplicating the
remove code paths.

This patch also removes an unnecessary check around comparing the hash
of the stored data to the new data's hash. That check can't fail since
the hash was a requirement for the map lookup in the first place.
Make the remove validation more robust by asserting that the
correct remove message is broadcast. This will provide a better
safety net when combining the remove functions.
Now that all the code is abstracted and tested, the remove()
and removeMailboxData() functions are identical. Combine them and update
callers appropriately.

Now, any caller don't need to know the difference and it removes the
sharp edge originally found in #3556
Use a more compact version of string formatting
in log messages

Rename isMetadataEquals to matchesRelevantPubKey
which is more descriptive of the actual check
@julianknutsen julianknutsen force-pushed the julianknutsen:entry-stubs-test branch from 0ab156f to 40340ee Nov 13, 2019
Now that the unit tests cover all of the per-Entry validation,
the tests that create specific configuration of ProtectedStorageEntry
and ProtectedMailboxStorageEntry objects can be removed in favor
of mockable Entrys.

Using mocks prior to this patch was impossible due to the relationship
between the Entry objects and the P2PDataStorage helper functions. Now
that the objects are properly abstracted and tested, real unit tests
can be written for the P2PDataStore module.

This patch leaves the tests and adds an @ignore so the reviewer can see
which unit test now supersedes the integration test.
Purge all the superseded tests and helper functions now that everything
can be unit tested.
@julianknutsen julianknutsen force-pushed the julianknutsen:entry-stubs-test branch from 40340ee to c652528 Nov 13, 2019
Copy link
Member

freimair left a comment

utAck. Waiting for comments in previous PRs to be addressed.

@freimair freimair mentioned this pull request Nov 15, 2019
@julianknutsen julianknutsen mentioned this pull request Nov 17, 2019
Copy link
Member

freimair left a comment

Ack

Copy link
Member

ripcurlx left a comment

utACK

@ripcurlx ripcurlx merged commit b078e92 into bisq-network:master Nov 18, 2019
1 check passed
1 check passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@julianknutsen julianknutsen deleted the julianknutsen:entry-stubs-test branch Nov 18, 2019
@ripcurlx ripcurlx mentioned this pull request Dec 13, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.