- [#679] Implementation of the File-system based block cache

Closes #697
Closes #720
Closes #587
Closes #667
Closes #443

- [#790] Fix ShieldFundsTests

Closes #790
Removes comments on `ShieldFundsTests` since those issues have been fixed

Depends on zcash-light-client-ffi changes that adopt newer versions of
 librustzcash crates `zcash_primitives 0.10`, `zcash_client_backend 0.7`,
`zcash_proofs 0.10`, `zcash_client_sqlite 0.5.0`.

Also allows wallets to define a shielding_threshold and will set
foundations to customize minimum confirmations for balances, spends
and shielding operations.

**Test Bootstrapping**

- `ZcashCompactBlockDescriptor`: struct that holds functions to
describe blocks as filenames and compare those filenames

`ZcashCompactBlockDescriptor.live` has the actual implementation
but it can be replaced by mocks if needed on Tests

main implementations are held under `FSCompactBlockRepository.filenameDescription` and `FSCompactBlockRepository.filenameComparison` on a separate extention

`DirectoryListingProviders` provide two default implementations of listing a
directory deterministically. `FileManager` does not define a sorting and needs to be done in-memory by calling `.sorted()` on the resulting collection. If this
is a big toll on performance it can be changed to a POSIX implementation but
this is good for now.

`ZcashCompactBlockDescriptor` adds a `height` helper function to
turn a filename into the height of the block stored.

Implemented `func latestHeight() throws -> BlockHeight ` that
returns the blockheight by querying the cache directory in a sorted
fashion and getting the last value and turning the filename into a
`BlockHeight`

Added `Meta` struct to ZcashCompactBlock.

Tests implemented:
- `filterBlockFiles`
- `testClearTheCache`
- `testLatestHeightEmptyCacheThrows`
- `testLatestHeightEmptyCacheThrowsAsync`
- `testRewindEmptyCacheDoesNothing`
- `testRewindEmptyCacheDoesNothingAsync`
- `testWhenBlockIsStoredItFollowsTheDescribedFormat`
- `testWhenBlockIsStoredItFollowsTheFilenameConvention`
- `testGetLatestHeight`
- `testRewindDeletesTheRightBlocks` test
- `testPerformanceExample` test. This isn't a real performance test because the API doesn't work with async/await yet

adopts `shield_funds` shielding threshold parameter

Implements `initBlockMetadataDb` and fix tests
Renames dbCache parameter to `fsBlockDbRoot`. Builds but tests don't pass.
Removes cacheDb uses from code. Testing utilities still persist.
Added needed information in  MIGRATING and CHANGELOG.
Added helper to perform deletion of legacy db and creation a the
new file system backed cache.

Renames parameters and changes code where needed.
Network Constants turned into `enum` with static methods.
DeletelastDownloadedBlock helper from initializer
Removes CompactBlockStorage and CompactBlockEntity.
Implements `latestCachedBlockHeight` on rustbackend.

*Replaces dependencies on ZcashRustWelding with `FSMetadataStore`*

This allows the tests to not depend in a particular implementation
of either the MockRustBackend of or ZcashRustBackend. Also provides
a way to test errors properly and switch implementations of critical
areas like `writeBlocks`.

CHANGELOG.md

@@ -1,3 +1,49 @@
+# Unreleased
+
+## File system backed block cache
+
+File system based block cache. Compact blocks will now be stored
+on the file system. Caller must provide a `URL` pointing to the 
+filesystem root directory where the fsBlock cache is. this directory
+is expected to contain a `/blocks` sub-directory with the blocks stored
+in the convened filename format `{height}-{hash}-block`. This directory
+must be granted both read and write permissions.
+
+the file system cache will have a "bookkeeping" database that the rust
+welding layer will use to know the state of the cache and locate the 
+cached compact blocks. This directory can be deleted provided that the
+Compactblock processor or Synchronizer are not running. Upon deletion 
+caller is responsible for initializing these objects for the cache to 
+be created. 
+
+Implementation notes: Users of the SDK will know the path they will 
+provide but must assume nobehavior whatsoever or rely on the cached 
+information in any way since it is an internal state of the SDK. 
+Maintainers might provide no support for problems related to speculative
+use of the file system cache. If you consider your application needs any
+other information than the ones available through public APIs, please 
+file the corresponding feature request.
+
+### Added
+- `Initializer.migrateCacheDb()` helps you move to filesystem based 
+block cache
+
+- `Synchronizer.shieldFunds(spendingKey:memo:shieldingThreshold)` shieldingThreshold
+was added allowing wallets to manage their own shielding policies.
+
+### Removed
+- `InitializerError.cacheDbMigrationFailed`
+
+### Deprecations
+CacheDb references that were deprecated instead of **deleted** are pointing out
+that they should be useful for you to migrate from using cacheDb.
+
+- `ResourceProvider.cacheDbURL` deprecated but left for one release cycle for clients 
+to move away from cacheDb.
+
+- `NetworkConstants.defaultCacheDbName` deprecated but left for one release cycle for clients 
+to move away from cacheDb.
+
 # 0.18.0-beta
 
 ## Farewell Cocoapods.
@@ -42,10 +88,9 @@ Sources/ZcashLightClientKit/Resources/checkpoints/testnet/2200000.json
 - [#645] Default rewind after ReOrg is 20 blocks when it should be 10
     This fixes an issue where the default reorg was 20 blocks rewind instead of 10. The
 reorg count was incremented before calling the rewind height computing function.
-
-
-## Change in how we represent transactions. 
 
+## Use Librustzcash database views to query and represent transactions
+
 - [#556] Change data structures which represent transactions.
 
     These data types are gone: `Transaction`, `TransactionEntity`, `ConfirmedTransaction`, 
@@ -64,8 +109,7 @@ the `SDKSynchronizer` has now new methods to fetch those:
     - `func getMemos(for receivedTransaction: ZcashTransaction.Received) throws -> [Memo]`
     - `func getMemos(for sentTransaction: ZcashTransaction.Sent) throws -> [Memo]`
 
-## Compact Block Processor is not public anymore.
-
+## CompactBlockProcessor is now internal
 - [#671] Make CompactBlockProcessor Internal.
 
     The CompactBlockProcessor is no longer a public class/API. Any direct access will

Example/ZcashLightClientSample/ZcashLightClientSample.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved

@@ -140,8 +140,7 @@
       "kind" : "remoteSourceControl",
       "location" : "https://github.com/zcash-hackworks/zcash-light-client-ffi",
       "state" : {
-        "revision" : "b6013b8b313523b2c72ce62dbdc17f6ffad3a500",
-        "version" : "0.1.1"
+        "revision" : "4bf7225c4f668249661eb4b54dde3dd356a15365"
       }
     }
   ],

Example/ZcashLightClientSample/ZcashLightClientSample/AppDelegate.swift

@@ -5,7 +5,7 @@
 //  Created by Francisco Gindre on 06/09/2019.
 //  Copyright © 2019 Electric Coin Company. All rights reserved.
 //
-
+// swiftlint:disable force_cast force_try force_unwrapping
 import UIKit
 import ZcashLightClientKit
 import NotificationBubbles
@@ -38,7 +38,7 @@ class AppDelegate: UIResponder, UIApplicationDelegate {
                 .deriveFullViewingKey()
 
             let wallet = Initializer(
-                cacheDbURL: try! cacheDbURLHelper(),
+                fsBlockDbRoot: try! fsBlockDbRootURLHelper(),
                 dataDbURL: try! dataDbURLHelper(),
                 pendingDbURL: try! pendingDbURLHelper(),
                 endpoint: DemoAppConfig.endpoint,
@@ -84,7 +84,6 @@ class AppDelegate: UIResponder, UIApplicationDelegate {
 
     func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?) -> Bool {
         // Override point for customization after application launch.
-        _ = self.sharedWallet
         subscribeToMinedTxNotifications()
 
         return true
@@ -127,7 +126,13 @@ extension AppDelegate {
         } catch {
             loggerProxy.error("error clearing cache DB: \(error)")
         }
-
+
+        do {
+            try FileManager.default.removeItem(at: try fsBlockDbRootURLHelper())
+        } catch {
+            loggerProxy.error("error clearing FsBlockDBRoot: \(error)")
+        }
+
         do {
             try FileManager.default.removeItem(at: try dataDbURLHelper())
         } catch {
@@ -158,6 +163,15 @@ func documentsDirectoryHelper() throws -> URL {
     try FileManager.default.url(for: .documentDirectory, in: .userDomainMask, appropriateFor: nil, create: true)
 }
 
+func fsBlockDbRootURLHelper() throws -> URL {
+    try documentsDirectoryHelper()
+        .appendingPathComponent(kZcashNetwork.networkType.chainName)
+        .appendingPathComponent(
+            ZcashSDK.defaultFsCacheName,
+            isDirectory: true
+        )
+}
+
 func cacheDbURLHelper() throws -> URL {
     try documentsDirectoryHelper()
         .appendingPathComponent(

Example/ZcashLightClientSample/ZcashLightClientSample/Get UTXOs/GetUTXOsViewController.swift

@@ -49,7 +49,8 @@ class GetUTXOsViewController: UIViewController {
             Task { @MainActor in
                 let transaction = try await AppDelegate.shared.sharedSynchronizer.shieldFunds(
                     spendingKey: usk,
-                    memo: try Memo(string: "shielding is fun!")
+                    memo: try Memo(string: "shielding is fun!"),
+                    shieldingThreshold: Zatoshi(10000)
                 )
                 KRProgressHUD.dismiss()
                 self.messageLabel.text = "funds shielded \(transaction)"

MIGRATING.md

@@ -1,3 +1,47 @@
+# Migrating from previous versions to <Unreleased>
+Compact block cache no longer uses a sqlite database. The existing database
+should be deleted. `Initializer` now takes an `fsBlockDbRootURL` which is a 
+URL pointing to a RW directory in the filesystem that will be used to store
+the cached blocks and the companion database managed internally by the SDK.
+
+`Initializer` provides a convenience initializer that takes the an optional
+URL to the `cacheDb` location to migrate the internal state of the 
+`CompactBlockProcessor` and delete that database. 
+
+````Swift
+    convenience public init (
+        cacheDbURL: URL?,
+        fsBlockDbRoot: URL,
+        dataDbURL: URL,
+        pendingDbURL: URL,
+        endpoint: LightWalletEndpoint,
+        network: ZcashNetwork,
+        spendParamsURL: URL,
+        outputParamsURL: URL,
+        viewingKeys: [UnifiedFullViewingKey],
+        walletBirthday: BlockHeight,
+        alias: String = "",
+        loggerProxy: Logger? = nil
+    )
+````
+
+We do not make any efforts to extract the cached blocks in the sqlite
+`cacheDb` and storing them on disk. Although this might be the logical 
+step to do, we think such migration as little to gain since a migration
+function will be a "run once" function with many different scenarios to
+consider and possibly very error prone. On the other hand, we rather delete
+the `cacheDb` altogether and free up that space on the users' devices since
+we have surveyed that the `cacheDb` as been growing exponentially taking up
+many gigabyte of disk space. We forsee that many possible attempts to copy
+information from one cache to another, would possibly fail 
+
+Consuming block cache information for other purposes is discouraged. Users
+must not make assumptions on its contents or rely on its contents in any way. 
+Maintainers assume that this state is internal and won't consider further
+uses other than the intended for the current development. If you consider
+your application needs any other information than the ones available through
+public APIs, please file the corresponding feature request.
+
 # Migrating from 0.16.x-beta to 0.17.0-alpha.x
 
 ## Changes to Demo APP
@@ -78,4 +122,4 @@ async environment.
 
 SDK Clients will likely be affected by some `async` methods on `SDKSynchronizer`.
 
-We recommend clients that don't support structured concurrency features, to work around this by  surrounding the these function calls either in @MainActor contexts either by marking callers as @MainActor or launching tasks on that actor with `Task { @MainActor in ... }`
+We recommend clients that don't support structured concurrency features, to work around this by  surrounding the these function calls either in @MainActor contexts either by marking callers as @MainActor or launching tasks on that actor with `Task { @MainActor in ... }`

Package.resolved

@@ -86,8 +86,7 @@
       "kind" : "remoteSourceControl",
       "location" : "https://github.com/zcash-hackworks/zcash-light-client-ffi",
       "state" : {
-        "revision" : "b6013b8b313523b2c72ce62dbdc17f6ffad3a500",
-        "version" : "0.1.1"
+        "revision" : "ce66b2f1b354ff8eb192cf87866ef0f27a6f594f"
       }
     }
   ],

Package.swift

@@ -16,7 +16,7 @@ let package = Package(
     dependencies: [
         .package(url: "https://github.com/grpc/grpc-swift.git", from: "1.8.0"),
         .package(url: "https://github.com/stephencelis/SQLite.swift.git", from: "0.14.1"),
-        .package(name: "libzcashlc", url: "https://github.com/zcash-hackworks/zcash-light-client-ffi", from: "0.1.1")
+        .package(name: "libzcashlc", url: "https://github.com/zcash-hackworks/zcash-light-client-ffi", revision: "ce66b2f1b354ff8eb192cf87866ef0f27a6f594f")
     ],
     targets: [
         .target(
@@ -46,7 +46,8 @@ let package = Package(
                 .copy("Resources/cache.db"),
                 .copy("Resources/darkside_caches.db"),
                 .copy("Resources/darkside_data.db"),
-                .copy("Resources/darkside_pending.db")
+                .copy("Resources/darkside_pending.db"),
+                .copy("Resources/sandblasted_mainnet_block.json")
             ]
         ),
         .testTarget(