Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Live channel database backup #951

Merged
merged 17 commits into from Apr 19, 2019
Merged
Changes from 11 commits
Commits
File filter...
Filter file types
Jump to…
Jump to file or symbol
Failed to load files and symbols.

Always

Just for now

@@ -136,6 +136,18 @@ Eclair uses [`logback`](https://logback.qos.ch) for logging. To use a different
java -Dlogback.configurationFile=/path/to/logback-custom.xml -jar eclair-node-gui-<version>-<commit_id>.jar
```

#### Backup

The files that you need to backup are located in your data directory. You must backup:
- your seed (`seed.dat`)
- your channel database (`eclair.bak` under directory `mainnet`, `testnet` or `regtest` depending on which chain you're running on)

Your seed never changes once it is created, but your channels do change whenever you receive our send payments.
`eclair.bak` is safe to backup even when your system is running. We recommend that you implement your backup
process in 2 steps:
- first, rename `eclair.bak` to a new local file
This conversation was marked as resolved by pm47

This comment has been minimized.

Copy link
@pm47

pm47 Apr 19, 2019

Member
Suggested change
- first, rename `eclair.bak` to a new local file
- first, move `eclair.bak` to a new local file
- then, backup the renamed file using whatever tool you like.

## Docker

A [Dockerfile](Dockerfile) image is built on each commit on [docker hub](https://hub.docker.com/r/acinq/eclair) for running a dockerized eclair-node.
@@ -202,6 +214,9 @@ eclair.bitcoind.rpcuser=<your-mainnet-rpc-user-here>
eclair.bitcoind.rpcpassword=<your-mainnet-rpc-password-here>
```




## Resources
- [1] [The Bitcoin Lightning Network: Scalable Off-Chain Instant Payments](https://lightning.network/lightning-network-paper.pdf) by Joseph Poon and Thaddeus Dryja
- [2] [Reaching The Ground With Lightning](https://github.com/ElementsProject/lightning/raw/master/doc/deployable-lightning.pdf) by Rusty Russell
@@ -203,7 +203,7 @@
<dependency>
<groupId>org.xerial</groupId>
<artifactId>sqlite-jdbc</artifactId>
<version>3.21.0.1</version>
<version>3.27.2.1</version>
</dependency>
<dependency>
<!-- This is to get rid of '[WARNING] warning: Class javax.annotation.Nonnull not found - continuing with a stub.' compile errors -->
@@ -135,3 +135,10 @@ eclair {
private-key-file = "tor.dat"
}
}

// do not edit or move this section
eclair.backup-mailbox {
mailbox-type = "akka.dispatch.BoundedMailbox"
mailbox-capacity = 1
mailbox-push-timeout-time = 0
}
@@ -43,7 +43,7 @@ import fr.acinq.eclair.blockchain.fee.{ConstantFeeProvider, _}
import fr.acinq.eclair.blockchain.{EclairWallet, _}
import fr.acinq.eclair.channel.Register
import fr.acinq.eclair.crypto.LocalKeyManager
import fr.acinq.eclair.db.Databases
import fr.acinq.eclair.db.{BackupHandler, Databases}
import fr.acinq.eclair.io.{Authenticator, Server, Switchboard}
import fr.acinq.eclair.payment._
import fr.acinq.eclair.router._
@@ -88,11 +88,12 @@ class Setup(datadir: File,
val config = NodeParams.loadConfiguration(datadir, overrideDefaults)
val seed = seed_opt.getOrElse(NodeParams.getSeed(datadir))
val chain = config.getString("chain")
val chaindir = new File(datadir, chain)
val keyManager = new LocalKeyManager(seed, NodeParams.makeChainHash(chain))

val database = db match {
case Some(d) => d
case None => Databases.sqliteJDBC(new File(datadir, chain))
case None => Databases.sqliteJDBC(chaindir)
}

val nodeParams = NodeParams.makeNodeParams(config, keyManager, initTor(), database)
@@ -223,8 +224,6 @@ class Setup(datadir: File,
wallet = bitcoin match {
case Bitcoind(bitcoinClient) => new BitcoinCoreWallet(bitcoinClient)
case Electrum(electrumClient) =>
// TODO: DRY
val chaindir = new File(datadir, chain)
val sqlite = DriverManager.getConnection(s"jdbc:sqlite:${new File(chaindir, "wallet.sqlite")}")
val walletDb = new SqliteWalletDb(sqlite)
val electrumWallet = system.actorOf(ElectrumWallet.props(seed, electrumClient, ElectrumWallet.WalletParameters(nodeParams.chainHash, walletDb)), "electrum-wallet")
@@ -234,7 +233,8 @@ class Setup(datadir: File,
_ = wallet.getFinalAddress.map {
case address => logger.info(s"initial wallet address=$address")
}

// do not change the name of this actor. it is used in the configuration to specify a custom bounded mailbox
backupHandler = system.actorOf(SimpleSupervisor.props(BackupHandler.props(nodeParams.db, new File(chaindir, "eclair.bak")), "backuphandler", SupervisorStrategy.Resume))
audit = system.actorOf(SimpleSupervisor.props(Auditor.props(nodeParams), "auditor", SupervisorStrategy.Resume))
paymentHandler = system.actorOf(SimpleSupervisor.props(config.getString("payment-handler") match {
case "local" => LocalPaymentHandler.props(nodeParams)
@@ -0,0 +1,48 @@
package fr.acinq.eclair.db

import java.io.File

import akka.actor.{Actor, ActorLogging, Props}
import akka.dispatch.{BoundedMessageQueueSemantics, RequiresMessageQueue}
import fr.acinq.eclair.channel.ChannelPersisted


/**
* This actor will synchronously make a backup of the database it was initialized with whenever it receives
* a ChannelPersisted event.
* To avoid piling up messages and entering an endless backup loop, it is supposed to be used with a bounded mailbox
* with a single item:
*
* backup-mailbox {
This conversation was marked as resolved by pm47

This comment has been minimized.

Copy link
@pm47

pm47 Apr 18, 2019

Member

Isn't there a way to enforce this configuration? That seems a bit error prone.

Overall the pattern is pretty nice.

* mailbox-type = "akka.dispatch.BoundedMailbox"
* mailbox-capacity = 1
* mailbox-push-timeout-time = 0
* }
*
* Messages that cannot be processed will be sent to dead letters
*
* @param databases database to backup
* @param backupFile backup file
*
* Constructor is private so users will have to use BackupHandler.props() which always specific a custom mailbox
*/
class BackupHandler private(databases: Databases, backupFile: File) extends Actor with RequiresMessageQueue[BoundedMessageQueueSemantics] with ActorLogging {

// we listen to ChannelPersisted events, which will trigger a backup
context.system.eventStream.subscribe(self, classOf[ChannelPersisted])

def receive = {
case persisted: ChannelPersisted =>
val start = System.currentTimeMillis()
val tmpFile = new File(backupFile.getAbsolutePath.concat(".tmp"))
databases.backup(tmpFile)
val result = tmpFile.renameTo(backupFile)
require(result, s"cannot rename $tmpFile to $backupFile")
val end = System.currentTimeMillis()
log.info(s"database backup triggered by channelId=${persisted.channelId} took ${end - start}ms")
}
}

object BackupHandler {
def props(databases: Databases, backupFile: File) = Props(new BackupHandler(databases, backupFile)).withMailbox("eclair.backup-mailbox")
}
@@ -19,6 +19,7 @@ trait Databases {

val pendingRelay: PendingRelayDb

def backup(file: File) : Unit
}

object Databases {
@@ -45,6 +46,12 @@ object Databases {
override val peers = new SqlitePeersDb(eclairJdbc)
override val payments = new SqlitePaymentsDb(eclairJdbc)
override val pendingRelay = new SqlitePendingRelayDb(eclairJdbc)
override def backup(file: File): Unit = {
This conversation was marked as resolved by pm47

This comment has been minimized.

Copy link
@pm47

pm47 Apr 17, 2019

Member

This API makes the assumption that we will only use a single file to store the databases, which is a bit strange.

SqliteUtils.using(eclairJdbc.createStatement()) {
statement => {
statement.executeUpdate(s"backup to ${file.getAbsolutePath}")
}
}
}
}

}
@@ -82,7 +82,7 @@ object SqliteUtils {
* Obtain an exclusive lock on a sqlite database. This is useful when we want to make sure that only one process
* accesses the database file (see https://www.sqlite.org/pragma.html).
*
* The lock will be kept until the database is closed, or if the locking mode is explicitely reset.
* The lock will be kept until the database is closed, or if the locking mode is explicitly reset.
*
* @param sqlite
*/
@@ -0,0 +1,38 @@
package fr.acinq.eclair.db

import java.io.File
import java.sql.DriverManager
import java.util.UUID

import akka.actor.{ActorSystem, Props}
import akka.testkit.TestKit
import fr.acinq.eclair.channel.ChannelPersisted
import fr.acinq.eclair.db.sqlite.SqliteChannelsDb
import fr.acinq.eclair.{TestConstants, TestUtils, randomBytes32}
import org.scalatest.FunSuiteLike

import scala.concurrent.duration._

class BackupHandlerSpec extends TestKit(ActorSystem("test")) with FunSuiteLike {

test("process backups") {
val db = TestConstants.inMemoryDb()
val wip = new File(TestUtils.BUILD_DIRECTORY, s"wip-${UUID.randomUUID()}")
val dest = new File(TestUtils.BUILD_DIRECTORY, s"backup-${UUID.randomUUID()}")
wip.deleteOnExit()
dest.deleteOnExit()
val channel = ChannelStateSpec.normal
db.channels.addOrUpdateChannel(channel)
assert(db.channels.listLocalChannels() == Seq(channel))

val handler = system.actorOf(BackupHandler.props(db, dest))
handler ! ChannelPersisted(null, TestConstants.Alice.nodeParams.nodeId, randomBytes32, null)
handler ! ChannelPersisted(null, TestConstants.Alice.nodeParams.nodeId, randomBytes32, null)
handler ! ChannelPersisted(null, TestConstants.Alice.nodeParams.nodeId, randomBytes32, null)
awaitCond(dest.exists(), 5 seconds)

val db1 = new SqliteChannelsDb(DriverManager.getConnection(s"jdbc:sqlite:$dest"))
val check = db1.listLocalChannels()
assert(check == Seq(channel))
}
}
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.