Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a proper payments database #885

Merged
merged 93 commits into from Apr 16, 2019

Conversation

Projects
None yet
6 participants
@pm47
Copy link
Member

commented Mar 7, 2019

This builds on #884.

There is no unique identifier for payments in LN protocol. Critically,
we can't use payment_hash as a unique id because there is no way to
ensure unicity at the protocol level.

Also, the general case for a "payment" is to be associated to multiple
update_add_htlcs, because of automated retries. We also routinely
retry payments, which means that the same payment_hash will be
conceptually linked to a list of lists of update_add_htlcs.

In order to address this, we introduce a payment id, which uniquely
identifies a payment, as in a set of sequential update_add_htlc
managed by a single PaymentLifecycle that ends with a PaymentSent or
PaymentFailed outcome.

We can then query the api using either payment_id or payment_hash.
The former will return a single payment status, the latter will return a
set of payment statuses, each identified by their payment_id.

add a payment identifier
This is in preparation for a proper payments database.

There is no unique identifier for payments in LN protocol. Critically,
we can't use `payment_hash` as a unique id because there is no way to
ensure unicity at the protocol level.

Also, the generatl case for a "payment" is to be associated to multiple
`update_add_htlc`s, because of automated retries. We also routinely
retry payments, which means that the same `payment_hash` will be
conceptually linked to a list of lists of `update_add_htlc`s.

In order to address this, we introduce a payment id, which uniquely
identifies a payment, as-in a set of sequential `update_add_htlc`
managed by a single `PaymentLifecycle` that ends with a `PaymentSent` or
`PaymentFailed` outcome.

We can then query the api using either `payment_id` or `payment_hash`.
The former will return a single payment status, the latter will return a
set of payment statuses, each identified by their `payment_id`.

NB: This is just the first step, there is no payment db yet and the ids
are not actually used. Some db tests do not pass.

@pm47 pm47 changed the title Add a proper payments database [WIP] Add a proper payments database Mar 7, 2019

@pm47

This comment has been minimized.

Copy link
Member Author

commented Mar 7, 2019

NB: this is best reviewed using a diff against #884.

pm47 and others added some commits Mar 7, 2019

add a payment identifier
This is in preparation for a proper payments database.

There is no unique identifier for payments in LN protocol. Critically,
we can't use `payment_hash` as a unique id because there is no way to
ensure unicity at the protocol level.

Also, the generatl case for a "payment" is to be associated to multiple
`update_add_htlc`s, because of automated retries. We also routinely
retry payments, which means that the same `payment_hash` will be
conceptually linked to a list of lists of `update_add_htlc`s.

In order to address this, we introduce a payment id, which uniquely
identifies a payment, as-in a set of sequential `update_add_htlc`
managed by a single `PaymentLifecycle` that ends with a `PaymentSent` or
`PaymentFailed` outcome.

We can then query the api using either `payment_id` or `payment_hash`.
The former will return a single payment status, the latter will return a
set of payment statuses, each identified by their `payment_id`.

NB: This is just the first step, there is no payment db yet and the ids
are not actually used. Some db tests do not pass.

# Conflicts:
#	eclair-core/src/main/scala/fr/acinq/eclair/channel/ChannelTypes.scala
#	eclair-core/src/main/scala/fr/acinq/eclair/db/sqlite/SqliteAuditDb.scala
#	eclair-core/src/main/scala/fr/acinq/eclair/payment/PaymentEvents.scala
#	eclair-core/src/main/scala/fr/acinq/eclair/payment/PaymentLifecycle.scala
#	eclair-core/src/main/scala/fr/acinq/eclair/payment/Relayer.scala
#	eclair-core/src/test/scala/fr/acinq/eclair/api/JsonSerializersSpec.scala
#	eclair-core/src/test/scala/fr/acinq/eclair/channel/states/StateTestsHelperMethods.scala
#	eclair-core/src/test/scala/fr/acinq/eclair/channel/states/e/NormalStateSpec.scala
#	eclair-core/src/test/scala/fr/acinq/eclair/db/ChannelStateSpec.scala
#	eclair-core/src/test/scala/fr/acinq/eclair/db/SqliteAuditDbSpec.scala
#	eclair-core/src/test/scala/fr/acinq/eclair/payment/PaymentLifecycleSpec.scala
#	eclair-core/src/test/scala/fr/acinq/eclair/payment/RelayerSpec.scala
#	eclair-core/src/test/scala/fr/acinq/eclair/wire/ChannelCodecsSpec.scala
Merge branch 'payments-db-extended' into payments-db
# Conflicts:
#	eclair-core/src/main/scala/fr/acinq/eclair/channel/Channel.scala
#	eclair-core/src/main/scala/fr/acinq/eclair/channel/ChannelTypes.scala
#	eclair-core/src/main/scala/fr/acinq/eclair/db/sqlite/SqliteAuditDb.scala
#	eclair-core/src/main/scala/fr/acinq/eclair/payment/PaymentEvents.scala
#	eclair-core/src/main/scala/fr/acinq/eclair/payment/PaymentLifecycle.scala
#	eclair-core/src/main/scala/fr/acinq/eclair/payment/Relayer.scala
#	eclair-core/src/test/scala/fr/acinq/eclair/api/JsonSerializersSpec.scala
#	eclair-core/src/test/scala/fr/acinq/eclair/channel/states/e/NormalStateSpec.scala
#	eclair-core/src/test/scala/fr/acinq/eclair/channel/states/f/ShutdownStateSpec.scala
#	eclair-core/src/test/scala/fr/acinq/eclair/channel/states/g/NegotiatingStateSpec.scala
#	eclair-core/src/test/scala/fr/acinq/eclair/channel/states/h/ClosingStateSpec.scala
#	eclair-core/src/test/scala/fr/acinq/eclair/db/ChannelStateSpec.scala
#	eclair-core/src/test/scala/fr/acinq/eclair/db/SqliteAuditDbSpec.scala
#	eclair-core/src/test/scala/fr/acinq/eclair/payment/ChannelSelectionSpec.scala
#	eclair-core/src/test/scala/fr/acinq/eclair/payment/PaymentLifecycleSpec.scala
#	eclair-core/src/test/scala/fr/acinq/eclair/payment/RelayerSpec.scala
#	eclair-core/src/test/scala/fr/acinq/eclair/wire/ChannelCodecsSpec.scala
@araspitzu

This comment has been minimized.

Copy link
Member

commented Apr 2, 2019

The /send API now immediately returns a UUID, outgoing payments are stored in the paymentsDB and associated with a status, the status is updated every time there is a change by the payment lifecycle. There is also a /paymentinfo API that can be queried via payment-id and returns its status along with some extra info.

araspitzu added some commits Apr 3, 2019

Merge branch 'master' into payments-db
# Conflicts:
#	eclair-core/src/main/scala/fr/acinq/eclair/Eclair.scala
#	eclair-core/src/main/scala/fr/acinq/eclair/channel/Helpers.scala
@araspitzu

This comment has been minimized.

Copy link
Member

commented Apr 3, 2019

The PR also adds an optional fallback address to the /receive API, see b43ec23

@araspitzu

This comment has been minimized.

Copy link
Member

commented Apr 3, 2019

TODO: update the doc

@araspitzu araspitzu changed the title [WIP] Add a proper payments database Add a proper payments database Apr 3, 2019

pm47 and others added some commits Apr 12, 2019

@pm47

This comment has been minimized.

Copy link
Member Author

commented Apr 12, 2019

Serialization result is weird. Why the nulls?

{
    "prefix": "lntb",
    "amount": null,
    "timestamp": 1555089352,
    "nodeId": "03dca9db59c25bfb430667d5c080b55a08b808b39251ec9e5b3d86939262d18e6a",
    "description": "test",
    "paymentHash": "b80ea80ec5d6d1182ea0fa418c7c02ddac3e7d6af686449a46450712155ba2a5",
    "expiry": 3600,
    "minFinalCltvExpiry": null,
    "serialized": "lntb1pwtp37gpp5hq82srk96mg3st4qlfqcclqzmkkrult276ryfxjxg5r3y92m52jsdq8w3jhxaqxqrrssf8gdlty0v67dq3xn6m4e6je84ym65p5uf3a5pmqrlmdyk0ch93ak2pndqhqf5khswq32wmhwq0xyc22qpx4f0zpk77nvq55s37x5sdqqqfwpkw"
}
@n1bor

This comment has been minimized.

Copy link
Contributor

commented Apr 12, 2019

my issues all fixed.

One other point though sentinfo does not return the pre-image? Looks like it is missing from the OutgoingPayment class?
The pre-image is obviously really useful to prove you made the payment. Is in the GUI just not the sent_payments table.

araspitzu added some commits Apr 15, 2019

araspitzu added some commits Apr 15, 2019

Merge branch 'master' into payments-db
# Conflicts:
#	eclair-core/src/main/scala/fr/acinq/eclair/Eclair.scala
#	eclair-core/src/main/scala/fr/acinq/eclair/api/Service.scala
#	eclair-core/src/test/scala/fr/acinq/eclair/api/ApiServiceSpec.scala
#	eclair-core/src/test/scala/fr/acinq/eclair/integration/IntegrationSpec.scala
@pm47
Copy link
Member Author

left a comment

This LGTM provided that we increase the test coverage with #938.

@pm47 pm47 requested a review from sstone Apr 15, 2019

@pm47 pm47 assigned sstone and unassigned pm47 Apr 15, 2019

@Kukks

This comment has been minimized.

Copy link
Contributor

commented Apr 15, 2019

Looking forward to this :D

statement.executeUpdate("CREATE TABLE IF NOT EXISTS payments (payment_hash BLOB NOT NULL PRIMARY KEY, amount_msat INTEGER NOT NULL, timestamp INTEGER NOT NULL)")
require(getVersion(statement, DB_NAME, CURRENT_VERSION) <= CURRENT_VERSION) // version 2 is "backward compatible" in the sense that it uses separate tables from version 1. There is no migration though
statement.executeUpdate("CREATE TABLE IF NOT EXISTS received_payments (payment_hash BLOB NOT NULL PRIMARY KEY, preimage BLOB NOT NULL, payment_request TEXT NOT NULL, received_msat INTEGER, created_at INTEGER NOT NULL, expire_at INTEGER, received_at INTEGER)")
statement.executeUpdate("CREATE TABLE IF NOT EXISTS sent_payments (id TEXT NOT NULL PRIMARY KEY, payment_hash BLOB NOT NULL, preimage BLOB, amount_msat INTEGER NOT NULL, created_at INTEGER NOT NULL, completed_at INTEGER, status VARCHAR NOT NULL)")

This comment has been minimized.

Copy link
@sstone

sstone Apr 16, 2019

Member

There should be an index on payment_hash too

This comment has been minimized.

Copy link
@araspitzu

araspitzu Apr 16, 2019

Member

Agreed, done in f7fc22c


override def listPaymentRequests(from: Long, to: Long): Seq[PaymentRequest] = listPaymentRequests(from, to, pendingOnly = false)

override def listPendingPaymentRequests(from: Long, to: Long): Seq[PaymentRequest] = listPaymentRequests(from, to, pendingOnly = true)

This comment has been minimized.

Copy link
@sstone

sstone Apr 16, 2019

Member

It would make sense to sort returned results by timestamp

This comment has been minimized.

Copy link
@araspitzu

araspitzu Apr 16, 2019

Member

Good one, done in 153b62c

araspitzu added some commits Apr 16, 2019

@sstone

sstone approved these changes Apr 16, 2019

@pm47 pm47 merged commit 9032da5 into master Apr 16, 2019

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@pm47 pm47 deleted the payments-db branch Apr 16, 2019

@Kukks Kukks referenced this pull request Apr 18, 2019

Closed

API Callback/Notifications #831

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.