Track p2p data store files using Git LFS

The large binary objects in p2p/src/main/resources/ are updated on every
Bisq release with the latest network data to avoid the need for new Bisq
clients to download all of this information from the network, which
would easily overload seed nodes and generally bog down the client.

This approach works well enough for its purposes, but comes with the
significant downside of storing all of this binary data in Git history
forever. The current version of these binary objects total about 65M,
and they grow with every release. In aggregate, this has caused the
total size of the repository to grow to 360M, making it cumbersome to
clone over a low-bandwith connection, and slowing down various local Git
operations.

To avoid further exacerbating this problem, this commit sets these files
up to be tracked via Git LFS. There's nothing we can do about the 360M
of files that already exist in history, but we can ensure it doesn't
grow in this unchecked way going forward. For an understanding of how
Git LFS works, see the reference material at [1], and see also the
sample project and README at [2].

The following command was used to track the files:

    $ git lfs track "p2p/src/main/resources/*BTC_MAINNET"
    Tracking "p2p/src/main/resources/AccountAgeWitnessStore_BTC_MAINNET"
    Tracking "p2p/src/main/resources/BlindVoteStore_BTC_MAINNET"
    Tracking "p2p/src/main/resources/DaoStateStore_BTC_MAINNET"
    Tracking "p2p/src/main/resources/ProposalStore_BTC_MAINNET"
    Tracking "p2p/src/main/resources/SignedWitnessStore_BTC_MAINNET"
    Tracking "p2p/src/main/resources/TradeStatistics2Store_BTC_MAINNET"

We are using GitHub's built-in LFS service here, and it's important to
understand that there are storage and bandwidth limits there. We have
1G total storage and 1G per month of bandwidth on the free tier. We will
certainly exceed this, and so must purchase at least one "data pack"
from GitHub, possibly two. One gets us to 50G storage and bandwith.

In an attempt to avoid unnecessary LFS bandwidth usage, this commit also
updates the Travis CI build configuration to cache Git LFS files, such
that they are not re-downloaded on every CI build (see [3] and [4]
below). With that out of the way, the variable determining whether we
exceed the monthly limit is how many clones we have every month, and
there are many, though it's not clear how many are are Travis CI and how
many are users / developers.

Tracking these files via LFS means that developers will need to have Git
LFS installed in order to properly synchronize the files. If a developer
does not have LFS installed, cloning will complete successfully and the
build would complete successfully, but the app would fail when trying to
actually load the p2p data store files. For this reason, the build has
been updated to proactively check that the p2p data store files have
been properly synchronized via LFS, and if not, the build fails with a
helpful error message. The docs/build.md instructions have also been
updated accordingly.

It is important that we make this change now, not only to avoid growing
the repository in the way described above as we have been doing now for
many releases, but also because we are now considering adding yet more
binary objects to the repository, as proposed at
bisq-network/projects#25.

[1]: https://git-lfs.github.com
[2]: https://github.com/cbeams/lfs-test
[3]: https://docs-staging.travis-ci.com/user/customizing-the-build/#git-lfs
[4]: travis-ci/travis-ci#8787 (comment)

.gitattributes

@@ -1,10 +1,8 @@
 # Auto detect text files and normalize line endings to LF
 # This will handle all files NOT found below
 * text=auto
-
 # These text files should retain Windows line endings (CRLF)
 *.bat           text eol=crlf
-
 # These binary files should be left untouched
 # (binary is a macro for -text -diff)
 *.bmp           binary
@@ -14,3 +12,4 @@
 *.jpg           binary
 *.jpeg          binary
 *.png           binary
+p2p/src/main/resources/*BTC_MAINNET filter=lfs diff=lfs merge=lfs -text

.travis.yml

@@ -3,8 +3,18 @@ jdk:
     - openjdk10
     - openjdk12
 
+cache:
+    directories:
+        - .git/lfs
+git:
+    lfs_skip_smudge: true
+
+install:
+    - git lfs pull
+
 before_install:
     grep -v '^#' assets/src/main/resources/META-INF/services/bisq.asset.Asset | sort --check --dictionary-order --ignore-case
+
 notifications:
     slack:
         on_success: change

build.gradle

@@ -270,6 +270,17 @@ configure(project(':p2p')) {
         testAnnotationProcessor "org.projectlombok:lombok:$lombokVersion"
         testCompile("org.mockito:mockito-core:$mockitoVersion")
     }
+
+    processResources.doFirst {
+        // Sanity check that Git LFS-managed data store files have actually been sync'd.
+        // If they have not, e.g. because Git LFS is not installed, they will be text files
+        // containing a sha256 hash of the remote object, indicating we should stop the
+        // build and inform the user how to fix the problem.
+        if (file('src/main/resources/ProposalStore_BTC_MAINNET').text.contains("oid sha256:"))
+            throw new GradleException("p2p data store files have not been synchronized. " +
+                "To fix this, ensure you have Git LFS installed and run `git lfs pull`. " +
+                "See docs/build.md for more information.")
+    }
 }
 
 

docs/build.md

@@ -1,6 +1,14 @@
 # Building Bisq
 
 
+## Install Git LFS
+
+Bisq uses Git LFS to track certain large binary files. Follow the instructions at https://git-lfs.github.com to install it, then run the following to command to verify the installation:
+
+    $ git lfs version
+    git-lfs/2.10.0 (GitHub; darwin amd64; go 1.13.6)
+
+
 ## Clone
 
     git clone https://github.com/bisq-network/bisq
@@ -20,7 +28,7 @@ If on Windows run `gradlew.bat build` instead.
 
 Bisq executables are now available in the root project directory. Run Bisq Desktop as follows:
 
-Note: bisq runs fine on jdk10 and jdk11. jdk12 is currently not supported. 
+Note: bisq runs fine on jdk10 and jdk11. jdk12 is currently not supported.
 
     ./bisq-desktop