Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
(3/6) [BUGFIX] #3629 (Duplicate payloads w/ 20-byte and 32-byte keys) #3636
New commits start at 849155a
Hotfix Needed: No
There was a bug when reconstructing the in-memory HashMap for
This PR cleans up the usages so that all outward users see the 32-byte key and all internal users can use the existing 20-byte key for persistence. This design is much safer than transitioning all persistent users to 32-byte keys and writing on-disk upgrade code.
All external listeners of Payloads don't look at the key. So, although they will receive 2 onAdded() calls instead of 1, they do the right thing by excluding the second payload since it is identical. I don't think there is any reason to publish a hotfix for this.
New Seednode <--> Old Client
Old Seednode <--> New Client
New Seednode <--> New Client
Instead of using a subclass that overwrites a value, utilize Guice to inject the real value of 10000 in the app and let the tests overwrite it with their own.
Remove unused imports and clean up some access modifiers now that the final test structure is complete
Previously, this interface was called each time an item was changed. This required listeners to understand performance implications of multiple adds or removes in a short time span. Instead, give each listener the ability to process a list of added or removed entrys which can help them avoid performance issues. This patch is just a refactor. Each listener is called once for each ProtectedStorageEntry. Future patches will change this.
Minor performance overhead for constructing MapEntry and Collections of one element, but keeps the code cleaner and all removes can still use the same logic to remove from map, delete from data store, signal listeners, etc. The MapEntry type is used instead of Pair since it will require less operations when this is eventually used in the removeExpiredEntries path.
…batch All current users still call this one-at-a-time. But, it gives the ability for the expire code path to remove in a batch.
This will cause HashMapChangedListeners to receive just one onRemoved() call for the expire work instead of multiple onRemoved() calls for each item. This required a bit of updating for the remove validation in tests so that it correctly compares onRemoved with multiple items.
…ch removes #3143 identified an issue that tempProposals listeners were being signaled once for each item that was removed during the P2PDataStore operation that expired old TempProposal objects. Some of the listeners are very expensive (ProposalListPresentation::updateLists()) which results in large UI performance issues. Now that the infrastructure is in place to receive updates from the P2PDataStore in a batch, the ProposalService can apply all of the removes received from the P2PDataStore at once. This results in only 1 onChanged() callback for each listener. The end result is that updateLists() is only called once and the performance problems are reduced. This removes the need for #3148 and those interfaces will be removed in the next patch.
freimair left a comment
I might be missing something here so here are some control questions to foster my understanding:
This behavior is only for
No, the client sends roughly 1kb of unnecessary 20-byte keys at startup for the
No, the seednode sends back around 50kb of duplicate
No, old clients use the 20-byte key for persistence. So after restart all
During initial creation of
It isn't that the new clients won't memorize it, it is that they only have in-memory representations using 32-byte keys now. So, until the seednode is updated to ONLY have 32-byte keys and NOT have 20-byte keys in the internal map, the seednodes will still send back the 20-byte key payloads. It isn't harmful because the duplicate paylaods sent back are ignored from the new client.
Addresses the first half of #3629 by ensuring that the reconstructed HashMap always has the 32-byte key for each payload. It turns out, the TempProposalStore persists the ProtectedStorageEntrys on-disk as a List and doesn't persist the key at all. Then, on reconstruction, it creates the 20-byte key for its internal map. The fix is to update the TempProposalStore to use the 32-byte key instead. This means that all writes, reads, and reconstrution of the TempProposalStore uses the 32-byte key which matches perfectly with the in-memory map of the P2PDataStorage that expects 32-byte keys. Important to note that until all seednodes receive this update, nodes will continue to have both the 20-byte and 32-byte keys in their HashMap.
After a deeper investigation of the persistence code, I found that TempProposals are stored to disk as a List and the Map.Entry<ByteArray, ProtectedStorageEntry> is actually reconstructed in-memory. So, I've updated e212240 accordingly with a much cleaner patch that just changes the 20-byte key to a 32-byte key.
freimair left a comment
Now is the time to do the change, as only
I updated the bug as well but wanted to include it here.
After more investigation, 455f7d2 also fixes an additional issue where we were retransmitting every non-persistable