Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Persist and republish mailbox messages #5072

Merged

Commits on Jan 12, 2021

  1. Copy the full SHA
    0656a52 View commit details
    Browse the repository at this point in the history
  2. Make some methods default

    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    73cfe0b View commit details
    Browse the repository at this point in the history
  3. Copy the full SHA
    a28a2f5 View commit details
    Browse the repository at this point in the history
  4. Copy the full SHA
    d470d73 View commit details
    Browse the repository at this point in the history
  5. Copy the full SHA
    ee3f158 View commit details
    Browse the repository at this point in the history
  6. Copy the full SHA
    33703c2 View commit details
    Browse the repository at this point in the history
  7. Copy the full SHA
    b8fc7f3 View commit details
    Browse the repository at this point in the history
  8. Cleanups

    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    4fc35a7 View commit details
    Browse the repository at this point in the history
  9. Copy the full SHA
    4c7b560 View commit details
    Browse the repository at this point in the history
  10. Refactor: Rename

    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    504fb2e View commit details
    Browse the repository at this point in the history
  11. Copy the full SHA
    cf8c93d View commit details
    Browse the repository at this point in the history
  12. Refactor: Rename

    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    79363e7 View commit details
    Browse the repository at this point in the history
  13. Remove nullable return value from decryptProtectedMailboxStorageEntry

    Remove Objects::nonNull check
    Split handleMailboxItem into 2 methods for persisting and
    for processing my mailbox msg
    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    63a0117 View commit details
    Browse the repository at this point in the history
  14. Add filter for isMine at getMyMailBoxMessages

    Add TODOs
    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    5245c20 View commit details
    Browse the repository at this point in the history
  15. Remove old code

    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    bfbc657 View commit details
    Browse the repository at this point in the history
  16. Add onRemoved method to remove mailbox entries once a remove message …

    …arrives
    
    Add republishMailBoxMessages method
    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    6146362 View commit details
    Browse the repository at this point in the history
  17. Persist map for removed mailbox messages (AddOncePayload more generally)

    We add the date when we add the hash so that we can remove expired data.
    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    32f8874 View commit details
    Browse the repository at this point in the history
  18. Copy the full SHA
    75e547f View commit details
    Browse the repository at this point in the history
  19. Refactor: rename method

    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    e69a3f6 View commit details
    Browse the repository at this point in the history
  20. Set addressPrefixHash to empty bytes.

    We have added the capability in 1.4.0 and have
    enforced 1.5.1 so no traders can use that old
    version anymore so the capability check is not
    needed anymore.
    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    1886b4e View commit details
    Browse the repository at this point in the history
  21. Add RemovedPayloadsStorageService

    Adjust param
    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    5ba7d78 View commit details
    Browse the repository at this point in the history
  22. Refactor: Rename class

    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    41a92ea View commit details
    Browse the repository at this point in the history
  23. Use a hashset for listeners

    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    9edad02 View commit details
    Browse the repository at this point in the history
  24. Use MailboxItem value at mailboxItemsByUid instead of a list:

    The uid is not perfect like a cryptographic hash but it can be
    considered safe enough to not have collusion. The list got filled
    with duplicates, so should have been a hashSet anyway.
    We cannot use the hash as that is not available at the remove method.
    We might refactor that in future to get rid of that problematic uid as
    key but will require a but more refactoring in the client code as well,
    as we do not pass around the outer envelope data but only the decrypted
    data.
    
    - Add protectedStorageEntry from persisted mailbox messages to
    P2PDataStorage at startup. This ensures that we add those keys to excludedKeys
    and that helps to reduce load for seed nodes at initial data response.
    
    - Refactoring at removeFromMapAndDataStore:
    
    - Add trace logs
    - Add more comments
    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    de131c8 View commit details
    Browse the repository at this point in the history
  25. Copy the full SHA
    a3b2aad View commit details
    Browse the repository at this point in the history
  26. Refactor: move duplicated method

    capabilityRequiredAndCapabilityNotSupported to CapabilityUtils
    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    026858d View commit details
    Browse the repository at this point in the history
  27. Add logs

    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    873b557 View commit details
    Browse the repository at this point in the history
  28. Set uid inside constructor

    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    7a2501e View commit details
    Browse the repository at this point in the history
  29. Add support for TTL defined by payload message so we can use

    different TTL for lower prio mailbox messages like AckMessages.
    
    As we cannot add a field without breaking signatures we
    need to use the extraMap in MailboxStoragePayload
    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    35ef3b8 View commit details
    Browse the repository at this point in the history
  30. Add comment

    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    659fe15 View commit details
    Browse the repository at this point in the history
  31. Refactor: Rename method

    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    320e5f1 View commit details
    Browse the repository at this point in the history
  32. Copy the full SHA
    a54813a View commit details
    Browse the repository at this point in the history
  33. Add republishExistingProtectedMailboxStorageEntry method

    Seed nodes republish the persisted mailbox messages after
    startup in chunks of 50 items with a 2 minute delay.
    
    Move check for hasSequenceNrIncreased in addProtectedStorageEntry
    earlier so we return earlier. This is a very common case for return
    if we receive outdated data (like republished mailbox data we have
    already received).
    
    Mark logs with ## for easier find/replace for dev testing...
    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    8e0181e View commit details
    Browse the repository at this point in the history
  34. Add temp. ignore to a failing test.

    By debugging I could not see why it fails, behaviour seems correct.
    I guess its a mock setup issue. Will look into it to fix it soon...
    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    e933bfd View commit details
    Browse the repository at this point in the history
  35. Copy the full SHA
    8d866d5 View commit details
    Browse the repository at this point in the history
  36. Refactor: return early

    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    a9802e6 View commit details
    Browse the repository at this point in the history
  37. Copy the full SHA
    d39a697 View commit details
    Browse the repository at this point in the history
  38. Fix incorrect comparison at RemovedPayloadsService.readPersisted

    Add comments
    Cleanups
    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    b4ab8a4 View commit details
    Browse the repository at this point in the history
  39. Add program argument flag republishMailboxEntries

    Only republish if that flag is set, not depending if its a seedNode
    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    337ea89 View commit details
    Browse the repository at this point in the history
  40. Add null check

    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    32aeda7 View commit details
    Browse the repository at this point in the history
  41. Copy the full SHA
    3875bd8 View commit details
    Browse the repository at this point in the history
  42. Copy the full SHA
    0f08794 View commit details
    Browse the repository at this point in the history
  43. Copy the full SHA
    bffe2b8 View commit details
    Browse the repository at this point in the history
  44. Copy the full SHA
    f536aba View commit details
    Browse the repository at this point in the history
  45. Copy the full SHA
    456ce70 View commit details
    Browse the repository at this point in the history
  46. Copy the full SHA
    2235c4f View commit details
    Browse the repository at this point in the history
  47. Refactor handleDecryptedMessageWithPubKey to use MailboxMessage

    instead of decryptedMessageWithPubKey.
    Here we change a bit the behaviour as now we check also if the
    TradeMessage is a MailboxMessage. As that method is only called
    from MailboxMessageService domain we never get a non MailboxMessage
    here anyway and it would have been a bug.
    
    Remove removeMailboxMsg(DecryptedMessageWithPubKey decryptedMessageWithPubKey) as not used anymore
    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    575196c View commit details
    Browse the repository at this point in the history
  48. Copy the full SHA
    5cdae86 View commit details
    Browse the repository at this point in the history
  49. Refactor: rename getMyDecryptedMessages method to make it more explicit

    that its only used for MailboxMsg
    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    d353140 View commit details
    Browse the repository at this point in the history
  50. Copy the full SHA
    c229c3f View commit details
    Browse the repository at this point in the history
  51. Inline isMyMessage methods

    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    008fb3b View commit details
    Browse the repository at this point in the history
  52. Add containsKey check

    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    5c429bb View commit details
    Browse the repository at this point in the history
  53. Copy the full SHA
    4c003cd View commit details
    Browse the repository at this point in the history
  54. Remove outdated code

    Remove outdated comment
    Add checkNotNull for senderNodeAddress
    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    46f6026 View commit details
    Browse the repository at this point in the history
  55. Add TradeMailboxMessage and use a TTL of 15 days

    Extend TradeMailboxMessage where we had `extends TradeMessage implements MailboxMessage`
    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    470230c View commit details
    Browse the repository at this point in the history
  56. Let MailboxMessage extend ExpirablePayload:

    For all MailboxMessage we want to be sure they have an expire date.
    Add getTTL methods and TTL value.
    We use 7 days for ChatMessages and 30 days for PrivateNotificationMessages.
    All others use default 15 days.
    We prefer to keep them explicit and not use a default method in
    ExpirablePayload or MailboxMessage.
    The value in PrefixedSealedAndSignedMessage will not be used as the
    sender use the TTL of the payload and pass that to the MailboxStoragePayload
    where we store it in the extraMap. That way we can use different TTL even the
    payload message is encrypted and we would not be able to look it up.
    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    2b05ed5 View commit details
    Browse the repository at this point in the history
  57. Add toString

    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    364a7f5 View commit details
    Browse the repository at this point in the history
  58. Refactor: rename

    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    a21d79a View commit details
    Browse the repository at this point in the history
  59. Remove comment

    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    3e69ede View commit details
    Browse the repository at this point in the history
  60. Bugfix: Add processing of SendersNodeAddressMessage to

    onBundleOfEnvelopes method.
    
    In case we received a SendersNodeAddressMessage
    inside a BundleOfEnvelope we did not set the
    peers address.
    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    3834785 View commit details
    Browse the repository at this point in the history
  61. Copy the full SHA
    36657f5 View commit details
    Browse the repository at this point in the history
  62. Refactor: rename method

    chimp1984 committed Jan 12, 2021
    Copy the full SHA
    c12e239 View commit details
    Browse the repository at this point in the history
  63. Copy the full SHA
    2fb3bba View commit details
    Browse the repository at this point in the history

Commits on Jan 13, 2021

  1. Sort persisted mail box messages by age and limit total number.

    Add size limit.
    Add log for number of persisted mailbox msg per day
    chimp1984 committed Jan 13, 2021
    Copy the full SHA
    d434cb3 View commit details
    Browse the repository at this point in the history

Commits on Jan 14, 2021

  1. Copy the full SHA
    541f367 View commit details
    Browse the repository at this point in the history
  2. Copy the full SHA
    7ba5ab5 View commit details
    Browse the repository at this point in the history

Commits on Jan 15, 2021

  1. Return early

    chimp1984 committed Jan 15, 2021
    Copy the full SHA
    5d13fdc View commit details
    Browse the repository at this point in the history