lnd v0.7.0-beta
Roasbeef
Olaoluwa Osuntokun
This release marks a new major release of lnd that includes several important bug fixes, an improved payment API, pathfinding enhancements, faster initial sync times, support for fee bumping on sweeps, and an initial rollout of altruistic watchtowers. As always, users are highly encouraged to upgrade to this new version.
Database Migrations
This version includes two migrations, the first is in channel.db which modifies the structure of the payment tracking data to support the refactored router and its ability to reliably display payments via the RPC. The migration should look like this:
2019-06-14 21:54:53.576 [INF] LTND: Version: 0.7.0-beta commit=v0.7.0-beta, build=production, logging=default
2019-06-14 21:54:53.579 [INF] LTND: Active chain: Bitcoin (network=mainnet)
2019-06-14 21:54:53.583 [INF] CHDB: Checking for schema update: latest_version=9, db_version=8
2019-06-14 21:54:53.586 [INF] CHDB: Performing database schema migration
2019-06-14 21:54:53.586 [INF] CHDB: Applying migration #9
2019-06-14 21:54:53.586 [INF] CHDB: Migrating outgoing payments to new bucket structure
2019-06-14 21:54:53.586 [INF] CHDB: Migration of outgoing payment bucket structure completed!
The second migration is in wallet.db, which prunes redundant block data already being stored by the underlying backend. The migration should look like this:
2019-06-14 21:54:53.744 [INF] LNWL: Applying wallet address manager migration #8
2019-06-14 21:54:53.746 [INF] LNWL: Removing block hash entries beyond maximum reorg depth of 10000 from current tip 580748
Once updated, it will not be possible to return to an older version of lnd.
Verifying the Release
In order to verify the release, you'll need to have gpg or gpg2 installed on your system. Once you've obtained a copy (and hopefully verified that as well), you'll first need to import the keys that have signed this release if you haven't done so already:
curl https://keybase.io/roasbeef/pgp_keys.asc | gpg --import
Once you have the required PGP keys, you can verify the release (assuming manifest-v0.7.0-beta.txt and manifest-v0.7.0-beta.txt.sig are in the current directory) with:
gpg --verify manifest-v0.7.0-beta.txt.sig
You should see the following if the verification was successful:
gpg: assuming signed data in 'manifest-v0.7.0-beta.txt'
gpg: Signature made Tue Jul 2 09:46:47 2019 PDT
gpg: using RSA key F8037E70C12C7A263C032508CE58F7F8E20FD9A2
gpg: Good signature from "Olaoluwa Osuntokun <laolu32@gmail.com>" [ultimate]
That will verify the signature on the main manifest page which ensures integrity and authenticity of the binaries you've downloaded locally. Next, depending on your operating system you should then re-calculate the sha256 sum of the binary, and compare that with the following hashes (which are included in the manifest file):
f8452608ff2e3ca6a0327c830f2fee5e837b5a39e0bf7cedd190857a5b8894f8 lnd-darwin-386-v0.7.0-beta.tar.gz
57a2eef7c337dbad6bbdc2bdffe459292d34ce354b58d74e74b329753dc134c8 lnd-darwin-amd64-v0.7.0-beta.tar.gz
c7cd6a1f4980fbefaf4acc49940332d3159b7b6af989afb9d0c211abe0208d53 lnd-dragonfly-amd64-v0.7.0-beta.tar.gz
8380e5944053f5a8255deb0f69b3dc0a2bac30402b82abec9f348cad53ec166f lnd-freebsd-386-v0.7.0-beta.tar.gz
c818c3a983167312f3bf2c84cb285212c5052131319caaef287a97541d2ff479 lnd-freebsd-amd64-v0.7.0-beta.tar.gz
a1e861f4c9a4cf056030f40debd882c6f34502821d0edca27f415dcfbb9f7d8c lnd-freebsd-arm-v0.7.0-beta.tar.gz
47be6c3391fadbc5a169fa1dd6dd13031d759b3d42c71a2d556751746b705c48 lnd-linux-386-v0.7.0-beta.tar.gz
2e7ed105b9e57103645bda30501cbf3386909cfed19a2fabcc3dc9117ce99a8f lnd-linux-amd64-v0.7.0-beta.tar.gz
c995fa67d6b23e547723801de49817dda34188fba78d0fe8ae506774e54c0afd lnd-linux-arm64-v0.7.0-beta.tar.gz
a653b66e28b30131c9b766989cb490013128022e047273f287bf0f42a19693b9 lnd-linux-armv6-v0.7.0-beta.tar.gz
ac51d96ee9b57bfcab0b05dbcfcd9ce3bd42a216354c0972e97c1a1c86c2479a lnd-linux-armv7-v0.7.0-beta.tar.gz
a4a119855e3759e49472d7d0f1f8529e984e7e7fbcedb78463daf4d7f6aceb6e lnd-linux-mips64-v0.7.0-beta.tar.gz
2e06c33c0c8c4f6ef680071095e652ea0b32ff164545a60aa372de3b12644db1 lnd-linux-mips64le-v0.7.0-beta.tar.gz
6d4bc470ae424bf46f1057149880f60a83057f26693f0098f3e9dd774355cdcd lnd-linux-ppc64-v0.7.0-beta.tar.gz
d931981d6a742b07abc965e779fd8ed93a7dfdbbdeefea0a47b0cdf90a94b645 lnd-netbsd-386-v0.7.0-beta.tar.gz
db6d6be0cf5e7e791be097de944261db74ef1400b711dc6b825b21b7d3a2958f lnd-netbsd-amd64-v0.7.0-beta.tar.gz
0ef5470c6ba928e740bf83ba86b7af05eb0df0d3077e92c347fb93cdcf7fb276 lnd-openbsd-386-v0.7.0-beta.tar.gz
ec8dae2c01d818a6cbc622b07f5be8a4355a25cc7a887216edc232c528257c20 lnd-openbsd-amd64-v0.7.0-beta.tar.gz
d5d9178178dca9a3e770dc74d655f579e6aafaec9e7b32a726c44dc093c52aa0 lnd-source-v0.7.0-beta.tar.gz
254ccdf63c2dbd95381663be0e132d60f3423c9568d304a4384823c198d12f8a lnd-windows-386-v0.7.0-beta.zip
51badb5f690e8bc15e90331a42ea823399d1eb60708c4d682683f070ece13c23 lnd-windows-amd64-v0.7.0-beta.zip
4ee8e4b7d8372c8e750125dcdd93cd1b1b55687460a2f7fe1c8a23e60bb17e7b vendor.tar.gz
One can use the shasum -a 256 <file name here> tool in order to re-compute the sha256 hash of the target binary for your operating system. The produced hash should be compared with the hashes listed above and they should match exactly.
Finally, you can also verify the tag itself with the following command:
git verify-tag v0.7.0-beta
The binaries are compiled with go1.12.6 and include the following build tags: autopilotrpc, signrpc, walletrpc, chainrpc, invoicesrpc, and routerrpc.
Building the Contained Release
With this new version of lnd, we've modified our release process to ensure the bundled release is now fully self contained. As a result, with only the attached payload with this release, users will be able to rebuild the target release themselves without having to fetch any of the dependencies. Note that at this stage, binaries aren't yet fully reproducible (even with go modules). This is due to the fact that by default, Go will include the full directory path where the binary was built in the binary itself. As a result, unless your file system exactly mirrors the machine used to build the binary, you'll get a different binary, as it includes artifacts from your local file system. This will be fixed in go1.13, and before then we may modify our release system to do this automatically.
In order to re-build from scratch, assuming that vendor.tar.gz and lnd-source-v0.7.0-beta.tar.gz are in the current directory:
tar -xvzf vendor.tar.gz
tar -xvzf lnd-source-v0.7.0-beta.tar.gz
GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.7.0-beta" ./cmd/lnd
GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.7.0-beta" ./cmd/lncli
The -mod=vendor flag tells the go build command that it doesn't need to fetch the dependencies, and instead, they're all enclosed in the local vendor directory.
Additionally, it's now possible to use the enclosed release.sh script to bundle a release for a specific system like so:
LNDBUILDSYS="linux-arm64 darwin-amd64" ./release.sh
⚡️⚡️⚡️ OK, now to the rest of the release notes! ⚡️⚡️⚡️
Release Notes (in progress)
bitcoind 0.18 compatibility
This new version of lnd has updated the way we check for errors over the RPC interface after we broadcast a transaction to be compatible with bitcoind v0.18. Note that the “master” branch of bitcoind isn’t yet supported yet as it includes distinct RPC changes to sendrawtransaction in particular that will require further updates in lnd. We’re targeting lnd v0.7.1. to fix these remaining RPC compatibility issues.
Private Altruist Watchtowers
In this release, we’re rolling out the ability to run a private, altruist watchtower as a fully-integrated subsystem of lnd. Watchtowers act as a second line of defense in responding to malicious or accidental breach scenarios in the event that client’s node is offline or unable to respond at the time of a breach, offering greater degree of safety to channel funds.
In contrast to a reward watchtower which demand a portion of the channel funds as a reward for fulfilling its duty, an altruist watchtower returns all of the victim’s funds (minus on-chain fees) without taking a cut. Reward watchtowers will be enabled in a subsequent release, though are still undergoing further testing and refinement.
In addition, lnd can now be configured to operate as a watchtower client, backing up encrypted breach-remedy transactions (aka. justice transactions) to other altruist watchtowers. The watchtower stores the fixed-size, encrypted blobs and is only able to decrypt and publish the justice transaction after the offending party has broadcast a revoked commitment state. Client communications with a watchtower are encrypted and authenticated using ephemeral keypairs, mitigating the amount of tracking the watchtower can perform on its clients using long-term identifiers.
Note that we have chosen to deploy a restricted set of features in this release that can begin to provide meaningful security to lnd users. Many more watchtower-related features are nearly complete or have meaningful progress, and we will continue to ship them as they receive further testing and become safe to release.
Configuring a Watchtower
To set up a watchtower, command line users should compile in the optional watchtowerrpc subserver, which will offer the ability to interface with the tower via gRPC or lncli. The minimal configuration needed to activate the tower is watchtower.active=1.
By default, the watchtower will listen on :9911 which specifies port 9911 listening on all available interfaces. Users may configure their own listeners via the --watchtower.listen= option.
Additionally, users can specify their tower’s external IP address(es) using watchtower.externalips, which will expose the full tower URI over RPC. These can be given to clients in order to connect and use the tower.
Note: The watchtower’s public key is distinct from lnd’s node public key. For now this acts as a soft whitelist as it requires clients to know the tower’s public key in order to use it for backups before more advanced whitelisting features are implemented. We recommend NOT disclosing this public key openly, unless you are prepared to open your tower up to the entire Internet.
Retrieving information about your tower’s configurations can be done using lncli tower info:
🏔 lncli tower info
{
"pubkey": "03281d603b2c5e19b8893a484eb938d7377179a9ef1a6bca4c0bcbbfc291657b63",
"listeners": [
"[::]:9911"
],
"uris": [
"03281d603b2c5e19b8893a484eb938d7377179a9ef1a6bca4c0bcbbfc291657b63@1.2.3.4:9911"
}
The entire set of watchtower configuration options can be found using lncli -h:
watchtower:
--watchtower.active If the watchtower should be active or not
--watchtower.towerdir= Directory of the watchtower.db (default: $HOME/.lnd/data/watchtower)
--watchtower.listen= Add interfaces/ports to listen for peer connections
--watchtower.externalip= Add interfaces/ports where the watchtower can accept peer connections
--watchtower.readtimeout= Duration the watchtower server will wait for messages to be received before hanging up on client connections
--watchtower.writetimeout= Duration the watchtower server will wait for messages to be written before hanging up on client connections
Configuring a Watchtower Client
In order to set up a watchtower client, you’ll need the watchtower URI of an active watchtower. lnd can the ben configured to start using the tower by setting:
wtclient.private-tower-uris=03281d603b2c5e19b8893a484eb938d7377179a9ef1a6bca4c0bcbbfc291657b63@1.2.3.4:9911
Users may optionally configure the fee rate of justice transactions by setting the wtclient.sweep-fee-rate option, which accepts values in sat/byte.
The entire set of watchtower client configuration options can be found using lncli -h:
wtclient:
--wtclient.private-tower-uris= Specifies the URIs of private watchtowers to use in backing up revoked states. URIs must be of the form <pubkey>@<addr>. Only 1 URI is supported at this time, if none are provided the tower will not be enabled.
--wtclient.sweep-fee-rate= Specifies the fee rate in sat/byte to be used when constructing justice transactions sent to the watchtower.
For now, no information regarding the operation of the watchtower client is exposed over the RPC interface. We are working to expose this information in a later release, progress on this can be tracked in this PR. Users will be reliant on logs for observing the behavior of the client. We also plan to expand on the initial feature set by permitting multiple active towers for redundancy, as well as modifying the chosen set of towers dynamically without restarting the daemon.
gRPC Prometheus Exports
A build tag has been added: monitoring. If built with this tag, lnd has the ability to export Prometheus metrics about its gRPC performance. These metrics include keeping track of how many times each gRPC call has been invoked, how many bytes have been transmitted and received for each gRPC call, and more.
To enable Prometheus exporting after building lnd with the monitoring tag, run lnd with this flag:
--prometheus.enable
To change the address that lnd listens on for Prometheus, add this flag:
--prometheus.listen=<address>
By default, lnd's Prometheus exporter listens on localhost:8989.
Wallet Bug Fixes
A bug has been fixed that would cause the EstimateFee and SendOutputs RPC commands to fail if conditions of higher fees. The root cause was using the transactions effective fee rate instead of the minrelayfee when computing dust in certain scenarios.
Wallet Chain Sync Improvements
There was a period of time where the wallet would only store the latest 25,000 block headers of the chain in order to properly recover from chain reorgs, but this was modified to the existing behavior of storing all of them due to deep reorgs on the testnet chain during lnds initial testing and development. In this release, we revert back to the prior behavior, but will instead only store the latest 10,000 as a generous safety margin.
As a result of doing so, the initial sync process is sped up. Neutrino backends saw the least improvements in this department, while bitcoind saw the most.
The wallet will also now detect that the chain backend (i.e., btcd, bitcoind, etc.) is currently syncing if its latest block does not have a timestamp within the past 2 hours, allowing us to start lnd while the chain backend syncs.
UtxoSweeper Improvements
Recent versions of lnd have seen development of a new subsystem: the UtxoSweeper. The release notes of our v0.6.0-beta release provide a brief overview of its duties. In this release, we continue to expand its functionality by allowing users to obtain information about the list of inputs being batched by the UtxoSweeper through a new PendingSweeps RPC and its corresponding lncli wallet pendingseeps command.
We also now allow users to bump the fee rate of an input with the Replace-By-Fee (RBF) policy or a transaction with the Child-Pays-For-Parent (CPFP) policy through the BumpFee RPC and its corresponding lncli wallet bumpfee command. In order to provide this functionality, the UtxoSweeper first had to be extended to support sweeping distinct batches of inputs with similar fee preferences.
Fee bumping within lnd will take a different approach than the popular bitcoin-cli bumpfee command. Since the UtxoSweeper batches transactions together, lnd cannot rely on bumping a transaction’s fee by providing its hash since it can change at any point with the addition/removal of new inputs. Instead, fee bumping relies on an outpoint being provided along with a fee preference.
A fee preference can be defined as one of any of the following options, which will likely expand in future releases.
- Static fee rate: fixed fee rate to spent an input on-chain with.
- Confirmation target: a delta of blocks from the current height of the chain in which an input should be spend on-chain within.
When bumping the fee of an input that currently exists within the UtxoSweeper, a transaction is created with a fee rate backed by the fee preference provided that replaces the lower fee transaction through the RBF policy. When bumping the fee of an input that points to an unconfirmed transaction output which belongs to the user’s wallet, a transaction that spends this input is created with a fee rate backed by the fee preference provided. This spending transaction can be interpreted as the child in a CPFP scenario.
The current design presents some shortcomings which will be addressed in future releases as we develop things further. These include:
- Fee preference validation checks are not in place to ensure they provide a sufficiently higher fee rate in order for a fee bump to be successful. This responsibility is delegated to the user.
- The
UtxoSweeperdoes not persist any inputs to disk, so restartinglndwill require incomplete fee bump requests to be requested again.
lncli Bug Fixes
Config Changes
A new neutrino.assertfilterheader filter header flag has been added to lnd. In the past the neutrino protocol underwent a number of changes, some of which affected exactly what is included in the filter. Older clients may have been on persistent incorrect filter chains as a result of this. This new argument allows a client to ensure it has the right filter chain using an externally provided filter header hash. If this hash doesn’t match, then the neutrino client will delete their state and re-sync the filter header chain.
Litecoin support has been extended to also include the simnet and regtest chains.
Interoperability Improvements
CLTV Tie-Breaking in Commitment Output Sorting
A nuanced, spec-level bug has been fixed for properly sorting HTLCs on the commitment transaction. When HTLCs of identical payment hash and amount appear on the same commitment transaction, the original BIP69 sorting could lead to a channel failure if the HTLCs had different CLTV values since the second-layer transactions would have different sighashes. This happens because the CLTV is enforced via nLockTime field of the descendant transaction, and weren’t being considered in the sorting of the commitment outputs. The new spec sorting resolves this by using the CLTV values of HTLC outputs as tiebreakers when the payment hash and amount are identical.
Since 0.5, lnd has forbidden sending out more than one HTLC for the same payment hash at a given time, so users on pre-0.7 versions are unlikely to experience this bug in purely lnd to lnd interactions, or have triggered it on other nodes in the network. However, it’s possible that forwarding payments from implementations that permit duplicate payments can trigger the bug and result in a channel closure. This bug should be completely eliminated going forward.
Increased Non-initiator Funding Timeout
When another node initiates a channel open with lnd, the non-initiator forgets about the channel if it doesn’t confirm in timely manner. In 0.7, this timeout has been raised to two weeks from the prior value of two days. This allows lnd to tolerate channel initiators with a low time preference (set a low fee rate) and better tolerate fee spikes or on-chain congestion during the funding process.
P2P and Gossip Improvements
Prior versions of lnd would send out all updates (or new channels) to connected active syncing peers in a single payload. This could at times cause slight memory spikes if many channels were mined in a single block for example. This latest version of lnd will instead now split announcements into multi sub-batches when it’s time to flush the set of pending messages. This change reduces GC pressure, and also naturally implements a variant of flow control to ensure minimal load during announcement batch flushes. It also makes sure to redo a historical sync if a new peer has connected after losing all of its peers.
RPC Server and RPC Sub-Server Changes
lnd’s RPC certificates will now be automatically re-generated by the daemon if upon restart, we detect that they’ve expired. Many nodes that were created greater than a year ago may have run into this issue, and had to delete and then regenerate themselves.
The NodeUpdate and GetNodeInfo commands now include the color of the target node.
Transaction Hex in GetTransactions
The GetTransactions RPC call now also includes the serialized hex version of the transaction. This exposes information the wallet already has and can be helpful for external accounting systems connected to lnd, or for usage with the new BumpFee command added in this version.
Display Paid Payment Requests
When making payments via the SendPayment RPC, the paid invoice will now be displayed in the ListPayments results. Similarly, payments via lncli payinvoice will show the invoice in lncli listpayments. This allows applications to recover the details, e.g. the invoice memo, without needing to track them separately.
Zero Fees With UpdateChanPolicy
Channel Display in GetNodeInfo
The GetNodeInfo command will now optionally also return the set of channels belonging to a particular node. This is useful as it saves additional look ups, and a client-side mapping between channel ID and node using the DescribeGraph command. Arguments for the new lncli command are as follows:
⛰ lncli getnodeinfo -h
NAME:
lncli getnodeinfo - Get information on a specific node.
USAGE:
lncli getnodeinfo [command options] [arguments...]
CATEGORY:
Peers
DESCRIPTION:
Prints out the latest authenticated node state for an advertised node
OPTIONS:
--pub_key value the 33-byte hex-encoded compressed public of the target node
--include_channels if true, will return all known channels associated with the node
Structured SendToRoute Errors
A new SendToRoute rpc has been added to the router sub-server. This call improves upon the main rpc SendToRoute rpc by returning errors as proper proto messages. No more (regex) string parsing. As part of this change, the (deprecated) option to pass in multiple routes to the main SendToRoute was removed.
Payment RPCs
The interface of SendPayment in the router sub-server has been modified. It now takes a custom payment timeout value. This defines the duration after which no new attempts to complete the payment are initiated. The call also requires the fee limit to be set explicitly. The fee limit is an important parameter and not having a default value for this forces callers to make a conscious decision about its value. Another change is that SendPayment only returns payment-level failures. This set of failures is currently limited, but will be extended in the future. The previously returned last payment attempt error is no longer available, because its informative value is considered limited.
Furthermore a new call TrackPayment was added. TrackPayment can be used to pick up and track the status of pending payments, for example after losing the connection or across restarts.
Payment Persistence
A series of PRs (#3064, #3063, #2761, #2762) has been merged, making lnd better handle in-flights payments across restarts. lnd will now write any attempted payment to the database, such that should the daemon restart before the result comes back, it will correctly record it to the DB. This means that payments can now be in states “in flight”, “failed” or “succeeded”, which can all be shown by calling lncli listpayments --include_incomplete (#3190).
Probability Based Mission Control
In prior versions of lnd, when we encountered a path finding failure, we would simply ignore the edge or node that caused for error for 5 or 15 seconds, continuing one with that path finding session. This naive approach has a number of issues. As an example, this means that shortly after a node makes a payment, it forgets all the prior context it used to successfully route the payment. Additionally, the prior version didn’t share this information amongst concurrent in-flight payment sessions, meaning each session couldn’t learn from attempts of other on going sessions. Users may have noticed this “cold start” effect when they go to send a new payment a few minutes the last successful payment attempt. In older versions of lnd, the path finding would try a series of channels that it likely knew wouldn’t work, before finally finding the golden set of channels that reliably work.
In this new version of lnd, the previously used edge and node black lists in path finding are replaced by a probability based system. It modifies path finding so that it not only compares routes on fee and time lock, but also takes route success probability into account. Each channel starts out with a default “apriori” probability. As path finding attempts fail, we reduce the probability of channels that have failed to zero, exponentially decaying back to the default apiori probability. This means that lnd will now properly “learn” from its past path finding attempts. Additionally, since this algorithm is less greedy than the prior algorithm, it may return a distinct set of paths compared to the prior algorithm, exploring avenues that maybe didn’t seem “optimal”, but still have a high probability of success.
The new system is similar to “active learning” based approaches in machine learning. lnd will now continually update its view of the network based on empirical observations of payment failures. In addition to that, historical payment attempt outcomes are kept in mission control memory longer and improve path finding for future payments. A new diagnostic rpc has been added to query the current mission control state. Several configuration options are exposed to give more control of mission control behavior:
--routerrpc.minrtprob= Minimum required route success probability to attempt the payment (default: 0.01)
--routerrpc.apriorihopprob= Assumed success probability of a hop in a route when no other information is available. (default: 0.95)
--routerrpc.penaltyhalflife= Defines the duration after which a penalized node or channel is back at 50% probability (default: 1h0m0s)
--routerrpc.attemptcost= The (virtual) cost in sats of a failed payment attempt (default: 100)
A new querymc sub-server RPC command has been added to allow users to peek into this diagnostic information. Eventually users may be able to share their mission control state (the effective “memory” of the router), amongst their other nodes in order to ensure they don’t need to re-discover the set of reliable channels.
These changes prepare for further enhancements that will increase payment reliability.
The deprecated num_routes parameter of QueryRoutes has been removed.
Invoice Settling
Several changes with respect to invoice settling have been made. This makes the logic of off-chain and on-chain invoice settling more consistent and is a preparatory step for atomic multiple path payments.
Neutrino Improvements
All integration tests are now run continuously with the neutrino chain backend , in addition to the existing tests using btcd!
Static Channel Backups
A bug fix has been made to the current on-disk SCB format in the Windows operating system. Before this release, lnd wouldn’t properly close the temporary SCB file before swapping it over to the permanent file name (atomic file rename). On Windows in particular, this would cause the swap to fail, meaning that the file wouldn’t be updated on disk. We’ve fixed this issue in this latest release, by always closing the temporary file before we attempt to modify it.
Changelog
The full list of changes since 0.6.1-beta can be found here:
Contributors (Alphabetical Order)
2xic
AdamISZ
chokoboko
Conner Fromknecht
Daniel McNally
Eugene Zeigel
Federico Bond
Francisco Calderón
Geoff Taylor
Johan T. Halseth
John Griffith
Joost Jager
Matt Drollette
michael1011
Neevai Esinly
Olaoluwa Osuntokun
OrbitalTurtle
Valentine Wallace
Wilmer Paulino
Xavi Soler
Yaacov Akiba Slama