Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support resource files tagged by version to reduce initial data requests #4519

Commits on Sep 12, 2020

  1. Initial tests

    freimair committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    cb9f0f0 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    3296db3 View commit details
    Browse the repository at this point in the history
  3. Tests use static fixtures

    freimair committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    e7831d6 View commit details
    Browse the repository at this point in the history
  4. Migration test succeeds

    freimair committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    581286e View commit details
    Browse the repository at this point in the history
  5. GetMap test succeeds

    freimair committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    fc7a43d View commit details
    Browse the repository at this point in the history
  6. GetMapSinceFilter works

    freimair committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    d111d8c View commit details
    Browse the repository at this point in the history
  7. updateScenario works

    freimair committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    6eda9a9 View commit details
    Browse the repository at this point in the history
  8. freshInstall scenario works

    freimair committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    ff53f51 View commit details
    Browse the repository at this point in the history
  9. put scenario works

    freimair committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    49731f9 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    e123a0b View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    da5d41b View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    c7351a0 View commit details
    Browse the repository at this point in the history
  13. Refactored tests

    freimair committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    e2e80d7 View commit details
    Browse the repository at this point in the history
  14. Initial tests for requests

    freimair committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    10c1ecc View commit details
    Browse the repository at this point in the history
  15. Requests are smaller

    freimair committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    f6cea15 View commit details
    Browse the repository at this point in the history
  16. Configuration menu
    Copy the full SHA
    0d80123 View commit details
    Browse the repository at this point in the history
  17. Enforce version format and filter faulty ones

    Instead of trying to find faulty keys, we stick
    to the mechanics we have in place. This way,
    we have less false positives.
    
    However, we enforce a certain Bisq version
    format, as parsing and business logic relies
    on that now.
    freimair committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    aad57c3 View commit details
    Browse the repository at this point in the history
  18. Configuration menu
    Copy the full SHA
    faca8e4 View commit details
    Browse the repository at this point in the history
  19. Javadoc cosmetics

    freimair committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    72600b1 View commit details
    Browse the repository at this point in the history
  20. Init integration test script

    freimair committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    3ad395b View commit details
    Browse the repository at this point in the history
  21. Configuration menu
    Copy the full SHA
    2589137 View commit details
    Browse the repository at this point in the history
  22. Configuration menu
    Copy the full SHA
    34abad0 View commit details
    Browse the repository at this point in the history
  23. Use version history array

    Turns out, scanning resources does not work reliable enough. Thus,
    an array of Strings denoting historical Bisq versions it is.
    
    An other way to put it is it is an array denoting which data stores
    are there in the resources.
    freimair committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    62a77e3 View commit details
    Browse the repository at this point in the history
  24. Configuration menu
    Copy the full SHA
    39fcca5 View commit details
    Browse the repository at this point in the history
  25. Use full sync between seednodes

    If we use the diff sync between seed nodes we create a race condition
    where when a seednode gets updated to the new system, it does not
    sync up properly with the other seed nodes. And that would be fatal.
    
    So for the time being, when a seednode asks for data, it uses the
    "old" big requests with all object keys. Should not be a problem
    for now since they have enough bandwidth.
    freimair committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    a3628ff View commit details
    Browse the repository at this point in the history
  26. Configuration menu
    Copy the full SHA
    a374249 View commit details
    Browse the repository at this point in the history
  27. Ignore some tests

    CI does have troubles with tests which do file operations. Thus,
    these tests have been disabled. They have been useful during
    development and are useful for testing locally, though.
    freimair committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    8e55f64 View commit details
    Browse the repository at this point in the history
  28. Adjust P2PDataStorage tests

    freimair committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    e927e6d View commit details
    Browse the repository at this point in the history
  29. Add another migration test

    This test addresses the migration scenario where a user does not
    upgrade on the first possible occation (to the first Bisq version
    that has the new database structure in place) but does so later.
    freimair committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    2fb2232 View commit details
    Browse the repository at this point in the history
  30. Reactivate ignored tests

    freimair committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    5f8b7cf View commit details
    Browse the repository at this point in the history
  31. Refactoring and cleaning up

    freimair committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    f8320fe View commit details
    Browse the repository at this point in the history
  32. Cleanup

    freimair committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    017bbfd View commit details
    Browse the repository at this point in the history
  33. Refactored test helpers

    freimair committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    d2a6655 View commit details
    Browse the repository at this point in the history
  34. Fixed a potential issue when loading stores

    Previously, we only checked if the newest store is there. If it is
    not, we copy and reload everything. If it is there, we are satisfied.
    
    Now, we check for each and every store we should have and copy and
    load them individually.
    freimair committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    469c374 View commit details
    Browse the repository at this point in the history
  35. Fixed wrong filename

    freimair committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    249e185 View commit details
    Browse the repository at this point in the history
  36. Refactoring

    freimair committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    06ec223 View commit details
    Browse the repository at this point in the history
  37. Configuration menu
    Copy the full SHA
    eabadc0 View commit details
    Browse the repository at this point in the history
  38. Configuration menu
    Copy the full SHA
    755fc03 View commit details
    Browse the repository at this point in the history
  39. Renamed data stores

    freimair committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    d218f2e View commit details
    Browse the repository at this point in the history
  40. Replace TODO comment

    freimair committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    c34bfee View commit details
    Browse the repository at this point in the history
  41. Fixed a timing issue

    freimair committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    d6f195d View commit details
    Browse the repository at this point in the history
  42. Configuration menu
    Copy the full SHA
    d81bdab View commit details
    Browse the repository at this point in the history
  43. Update history version list

    freimair committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    abbfedc View commit details
    Browse the repository at this point in the history
  44. Removed deprecated test mechanisms

    Up until now, the tests needed to create a version history set for testing.
    These changes had to be made to a final static variable during runtime.
    Now, enough versions are in the history set anyways, so we could remove
    this part of the testing framework.
    freimair committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    5f94406 View commit details
    Browse the repository at this point in the history
  45. Cleanup imports

    freimair committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    0eccec7 View commit details
    Browse the repository at this point in the history
  46. Included PR feedback

    freimair committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    270f1e7 View commit details
    Browse the repository at this point in the history
  47. Included PR feedback

    freimair committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    c8c63ef View commit details
    Browse the repository at this point in the history
  48. Added 1.3.8 data stores

    freimair committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    76cc37e View commit details
    Browse the repository at this point in the history
  49. Use Version.VERSION again

    I thought I could get by using a fixed version string so we can
    have tests involing future data stores...
    freimair committed Sep 12, 2020
    Configuration menu
    Copy the full SHA
    b4aae51 View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2020

  1. Merge branch 'master_upstream' into chimp-reduce_initial_request_size

    # Conflicts:
    #	core/src/main/java/bisq/core/support/dispute/agent/MultipleHolderNameDetection.java
    #	core/src/main/java/bisq/core/trade/txproof/xmr/XmrTxProofService.java
    #	p2p/src/main/resources/AccountAgeWitnessStore_BTC_MAINNET
    #	p2p/src/main/resources/SignedWitnessStore_BTC_MAINNET
    #	p2p/src/main/resources/TradeStatistics2Store_BTC_MAINNET
    chimp1984 committed Sep 13, 2020
    Configuration menu
    Copy the full SHA
    f83c147 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    5f8e779 View commit details
    Browse the repository at this point in the history
  3. Use readStore(); instead of store = readStore(getFileName());

    storage.initAndGetPersisted and initAndGetPersistedWithFileName are not needed.
    
    Refactorings:
    - Extract variables for better readability
    - Rename variables
    - Split copyAndSplit to getStoreFromResource and pruneStore which describes better what is happening
    - Use getStore from base class, remove local readStore method
    chimp1984 committed Sep 13, 2020
    Configuration menu
    Copy the full SHA
    3100888 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    591c2fb View commit details
    Browse the repository at this point in the history
  5. Add mapOfHistoricalStores field

    Avoid that we have to iterate all historical stores at each getMap call and create that map once when the historical data are created.
    Reduce visibility
    Make fields final (readFromResources is not called twice)
    Rename variable for better readability
    chimp1984 committed Sep 13, 2020
    Configuration menu
    Copy the full SHA
    d5c6644 View commit details
    Browse the repository at this point in the history
  6. Revert critical data stores to current version.

    We should not risk to break those critical data and it does not provide that much benefit as with the trade statistics. Lets test it first with trade stats and if all goes well extend in a later release.
    chimp1984 committed Sep 13, 2020
    Configuration menu
    Copy the full SHA
    3dac4c4 View commit details
    Browse the repository at this point in the history
  7. Add version field

    chimp1984 committed Sep 13, 2020
    Configuration menu
    Copy the full SHA
    2cef411 View commit details
    Browse the repository at this point in the history
  8. Remove methods in MapStoreService and do instead instance of check in…

    … AppendOnlyDataStoreService to call the new method only where it makes sense.
    
    Remove the hack of adding a "magic" key for communicating the version. Use a new field in the request instead.
    
    Remove the extra handling of seedNode as the impl. did not make any difference  as the Version passed in
    `excludedKeys = getKeySetInBytes(this.appendOnlyDataStoreService.getMap("since " + Version.VERSION));` was same as in
    `if (!finalFilter.equals(Version.VERSION)) ` thus never doing anything there and the result maps is the same as in the seed node branch.
    
    Was not clear anyway what the motivation of a diff. handling is there? That seed nodes request always all data without filtering? Why?
    chimp1984 committed Sep 13, 2020
    Configuration menu
    Copy the full SHA
    8458a88 View commit details
    Browse the repository at this point in the history
  9. Add current version to version list. Makes it more clear that those a…

    …re all historical immutable data.
    
    Add ignoreHistoricalData param to getMap
    
    Remove unused variables/params
    chimp1984 committed Sep 13, 2020
    Configuration menu
    Copy the full SHA
    2ece512 View commit details
    Browse the repository at this point in the history
  10. Remove file

    chimp1984 committed Sep 13, 2020
    Configuration menu
    Copy the full SHA
    965de7f View commit details
    Browse the repository at this point in the history
  11. Rename SplitStore to PersistableNetworkPayload

    The store has nothing to do with the splitting of live data and historical data.
    It only provides a getMap method with the PersistableNetworkPayload type
    
    Use that for all stores with such a type of map.
    chimp1984 committed Sep 13, 2020
    Configuration menu
    Copy the full SHA
    cfefae2 View commit details
    Browse the repository at this point in the history
  12. Rename for better clarity

    chimp1984 committed Sep 13, 2020
    Configuration menu
    Copy the full SHA
    28987f5 View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    6cf054c View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    aadd2e4 View commit details
    Browse the repository at this point in the history
  15. Fix param in test. Add Ignore to the 2 test classes handling the new …

    …historical data store feature.
    
    @freimair Feel free to adjust the test to the new code.
    chimp1984 committed Sep 13, 2020
    Configuration menu
    Copy the full SHA
    7a5ba0b View commit details
    Browse the repository at this point in the history
  16. Fix param in test.

    chimp1984 committed Sep 13, 2020
    Configuration menu
    Copy the full SHA
    224eaab View commit details
    Browse the repository at this point in the history
  17. Revert changes at tests from prev version

    Some tests are still failing. Need to look into it another day...
    chimp1984 committed Sep 13, 2020
    Configuration menu
    Copy the full SHA
    98877b4 View commit details
    Browse the repository at this point in the history

Commits on Sep 15, 2020

  1. Merge branch 'master_upstream' into chimp-reduce_initial_request_size

    # Conflicts:
    #	build.gradle
    #	desktop/package/linux/Dockerfile
    #	desktop/package/linux/package.sh
    #	desktop/package/linux/release.sh
    #	desktop/package/macosx/create_app.sh
    #	desktop/package/macosx/finalize.sh
    #	desktop/package/windows/package.bat
    #	desktop/package/windows/release.bat
    #	relay/src/main/resources/version.txt
    chimp1984 committed Sep 15, 2020
    Configuration menu
    Copy the full SHA
    32cf32c View commit details
    Browse the repository at this point in the history
  2. Change dev mode price

    chimp1984 committed Sep 15, 2020
    Configuration menu
    Copy the full SHA
    8f2e4ca View commit details
    Browse the repository at this point in the history

Commits on Sep 16, 2020

  1. Configuration menu
    Copy the full SHA
    22fc97d View commit details
    Browse the repository at this point in the history
  2. Add getPersisted method

    chimp1984 committed Sep 16, 2020
    Configuration menu
    Copy the full SHA
    9c873a6 View commit details
    Browse the repository at this point in the history
  3. remove test files

    chimp1984 committed Sep 16, 2020
    Configuration menu
    Copy the full SHA
    3252a50 View commit details
    Browse the repository at this point in the history
  4. remove test

    chimp1984 committed Sep 16, 2020
    Configuration menu
    Copy the full SHA
    a3ea54a View commit details
    Browse the repository at this point in the history
  5. Fix issues and refactor

    chimp1984 committed Sep 16, 2020
    Configuration menu
    Copy the full SHA
    2b61563 View commit details
    Browse the repository at this point in the history
  6. Cleanup

    chimp1984 committed Sep 16, 2020
    Configuration menu
    Copy the full SHA
    996364e View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    4115c0c View commit details
    Browse the repository at this point in the history

Commits on Sep 17, 2020

  1. Configuration menu
    Copy the full SHA
    ad95019 View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2020

  1. Configuration menu
    Copy the full SHA
    e680c4d View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2020

  1. Configuration menu
    Copy the full SHA
    80bf8bf View commit details
    Browse the repository at this point in the history
  2. Merge branch 'chimp-reduce_initial_request_size' of https://github.co…

    …m/chimp1984/bisq into chimp-reduce_initial_request_size
    chimp1984 committed Sep 27, 2020
    Configuration menu
    Copy the full SHA
    3a4784e View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    c482a1b View commit details
    Browse the repository at this point in the history

Commits on Sep 28, 2020

  1. Fix mock class to reflect real class better.

    AppendOnlyDataStoreService does not have a map but aggregates the data form its services.
    chimp1984 committed Sep 28, 2020
    Configuration menu
    Copy the full SHA
    ba0dc1b View commit details
    Browse the repository at this point in the history
  2. Fix incorrect test.

    We use a set at the filter so it is not deterministic which item get truncated.
    chimp1984 committed Sep 28, 2020
    Configuration menu
    Copy the full SHA
    f132a35 View commit details
    Browse the repository at this point in the history
  3. Remove todo, add system line break

    If we get logs from windows users the missing line breaks makes it harder to read.
    chimp1984 committed Sep 28, 2020
    Configuration menu
    Copy the full SHA
    6eb1efb View commit details
    Browse the repository at this point in the history
  4. Add more logs

    chimp1984 committed Sep 28, 2020
    Configuration menu
    Copy the full SHA
    ee725bc View commit details
    Browse the repository at this point in the history

Commits on Sep 29, 2020

  1. Add isEmpty check

    chimp1984 committed Sep 29, 2020
    Configuration menu
    Copy the full SHA
    82c5b5e View commit details
    Browse the repository at this point in the history
  2. Do not call super.readFromResources(postFix); as we do not have a not…

    …-versioned resource file anymore. Instead use readStore(); to create the live store.
    
    Add more logs
    chimp1984 committed Sep 29, 2020
    Configuration menu
    Copy the full SHA
    171be24 View commit details
    Browse the repository at this point in the history
  3. Add more logs

    chimp1984 committed Sep 29, 2020
    Configuration menu
    Copy the full SHA
    bd38097 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    0feece6 View commit details
    Browse the repository at this point in the history
  5. Only call pruneStore if we have created a new historical store from r…

    …esources.
    
    Improve logs
    chimp1984 committed Sep 29, 2020
    Configuration menu
    Copy the full SHA
    e402c8a View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    620dfca View commit details
    Browse the repository at this point in the history
  7. Remove dev test data

    chimp1984 committed Sep 29, 2020
    Configuration menu
    Copy the full SHA
    90da37f View commit details
    Browse the repository at this point in the history
  8. Add more info in log

    chimp1984 committed Sep 29, 2020
    Configuration menu
    Copy the full SHA
    9240eec View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    ab82286 View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2020

  1. Merge branch 'master_upstream' into wip-merge

    # Conflicts:
    #	core/src/main/java/bisq/core/trade/statistics/TradeStatistics2Store.java
    #	p2p/src/main/java/bisq/network/p2p/network/Connection.java
    chimp1984 committed Oct 1, 2020
    Configuration menu
    Copy the full SHA
    f2cc5a6 View commit details
    Browse the repository at this point in the history