Refactor: Non-Interactive Provision Process #1126
Comments
|
Take care that also a re-factory fixes: |
|
This should be one of the firsts todo for a v1.7.1 .. because its needed as a prepartion for the v1.8 WebUI setup ans also before a possible c-lightning integration to keep merge conflicts low. It sould be started after project planning for v1.8 WebUI is done. |
|
A more detailed description of the non-interactive setup process can be found here: #43 (comment) |
|
Would like to add some relevant notes:
Also we can think ahead to be able to implement full disk encryption. All other passwords (for the LN wallet, bitcoinRPC, SSH login) can be randomized during the setup and shown securely over an encrypted tunnel like Tor / HTTPS / ssh. |
|
The disk encryption could be part of the boot from ssd feature #1235 - but that I see pushed back to v1.9 ... v1.8 is already a big package, we cannot do all at once. For this issue about the "Fast update option" I dont think we need a special feature. Am I correct that your your goal is to keep Lightning downtimes minimal to not decrease QualityOfService to network? If yes then I think with regard to this issue this can be done thru: In the non-interactive setup/recovery process make sure that the Bitcoin & Lightning services are started as early as possible to minimize downtime. Additional apps can install after that - this may slow down the update/recovery process but reducing downtime to the network is getting a bigger priority now that the network matures. |
|
Yes, agree that building the extra services after LND is started and unlocked is the best solution. Re. the disk encryption I wasn't thinking of implementing it just yet, but it is good to keep in mind how we could make it easier to be used in the future. |
rootzoll
changed the title
|
This is a first draft how non-interactive provision process could be part of the future v1.8 setup:
Source File: https://docs.google.com/drawings/d/1iYYXrZTFqH5xMVxSXP7BSmf38vIj6GhXtJoVGFCHrQE/edit?usp=sharing |
|
I know the words does not fit inside the box, but focus on the arrangements. |
|
@nyxnor sure you can see migration/setup dialogs as one part (thats either done by WebUI or SSH) but the detection if its a fresh/exiting/migrating blitz happens before the user interaction. OR hmmm .... But wait there is one point that the previous model maybe missed ... thats that I have for example a HDD from another fullnodeOS and I dont want to migrate - I just want to setup a raspiblitz and maybe just reuse the blockchain, or I have data of an old blitz on the HDD but like to start fresh, etc. So maybe I was missing a general step in the process, where after the But adding info on the Tor bridges can really happen so late in the process? The |
About missing general step, yeah, I understand this as a draft also, I missed some things too but that would add a lot of text and blocks for now.
It can be moved to the first acccess to an interface (ssh, or webui). |
|
I updated the draft .. as you can see the tor bridges should be able to set in the first
Source File: https://docs.google.com/drawings/d/1iYYXrZTFqH5xMVxSXP7BSmf38vIj6GhXtJoVGFCHrQE/edit?usp=sharing |
* move debug script * patch patch command * make sure setup file is sourced * remove debug output * make sure lnd is put behind tor * change indent * get fresh sync progress * always set password A * cached peer info * fix printing cache * fix check for existing files * handle bitcoind not running * result with newline * test line break * test new line * test new line * two vars on output * #2388 improve online check (less pinging) * used cached peer status * move chache * cach file permissions * allow sudo call * fix cache * remove double scan info * add conf info to sync screen * reorder info * add space * add space * order info * internet suppress error messages * order info * fix offering Blockchain copy * fix hostname * final ready state info * lnd unlock after provision * remove debug exit * harmonize ready state * add status to lnd unlock * update lnd unlock script * edit the unlock * remove debug echo * add debug * add debug * fix if statement * debug output * switch position of source setupdata
* fix copychain returns * typo in sync loop * stop services on inconsistent state * calling correct provisioning * apply bitcoin and lncli aliases in all scripts * network.aliases: add CLNETWORK * make cln default plugin dir: cln-plugins-enabled similar to the nginx model make 2 directories for plugins: cln-plugins-enabled - symlinked to ~/.lightning/plugins plugins from here are loaded automatically on cln start cln-plugins-available: plugins are downloaded here to be run until the next cln restart (or stopped with runonce) note the disk is mounted with noexec so plugins can't run from there discuss in: #2295 * move shutdown script * change all place where shutdown script is used * change notify & release * moved shutdown script * moved shutdown scripts * add more debug info * moving github script * remove chain in sync * no longer needed chain in sync * move debug script * patch patch command * make sure setup file is sourced * remove debug output * make sure lnd is put behind tor * change indent * get fresh sync progress * avoid scrolling in menus * use new selfsignedcert if no lnd tls.cert present * sparko: add info and connect menu with own cert #2295 * cln.rest: add connect option for Zeus #2295 * cln: add the backup plugin + options Usage options: cln-plugin.backup.sh [on] [testnet|mainnet|signet] cln-plugin.backup.sh [restore] [testnet|mainnet|signet] [force] cln-plugin.backup.sh [backup-compact] [testnet|mainnet|signet] https://github.com/lightningd/plugins/tree/master/backup Discussed in: #2295 * cln: add cln-plugin.standard.python.sh Install and show the output of the chosen plugin for C-lightning Usage: cln-plugin.standard-python.sh on [plugin-name] [testnet|mainnet|signet] [runonce] tested plugins: summary | helpme | feeadjuster find more at: https://github.com/lightningd/plugins discussed in: #2295 * shellcheck: change all `egrep` to `grep -E` https://github.com/koalaman/shellcheck/wiki/SC2196 * do not resolve aliases, use as variables * lnd: fix lnd.conf for parallel networks discussed in: #2290 * lnd: add LND option for parallel networks * deprecate Testnet in SETTINGS keysend and autopilot only for mainnet due to: #2290 * lnd: autopilot and autounlock for testnet * fix comments * add the SYSTEM menu for parallel chains * RTL update to v0.11.0 make chain specific directory for the config: /home/rtl/${netprefix}RTL/ use ${netprefix}lnd.conf in config override Environmen tvaribales for cln in the systemd service: /etc/systemd/system/${netprefix}${typeprefix}RTL.service discussed in: #2384 * lnd.setname.sh for testnet * display ${CHAIN} in the SYSTEM menu options * keep _aliases file when live patches are applied * all lncli_aliases to be used as variables * default to KIllMode=control-group in services https://www.man7.org/linux/man-pages/man5/systemd.kill.5.html discussed in: #1901 * add cln.hsmtool.sh for hsm_secret handling encrypt | decrypt | autounlock the hsm_secret for C-lightning usage: cln.hsmtool.sh [unlock] [testnet|mainnet|signet] cln.hsmtool.sh [encrypt|decrypt] [testnet|mainnet|signet] cln.hsmtool.sh [autounlock-on|autounlock-off] [testnet|mainnet|signet] discussed in: #2295 * add cln.install-service.sh to set up cln with systemd script to set up or update the CLN systemd service checks for hsm_secret encryption, autounlock and the sparko plugin usage: /home/admin/config.scripts/cln.install-service.sh $CHAIN discussed in: #2295 * use symlink to cln-plugins-enabled for all plugins * keep lnd autopilot and autounlock mainnet only mainnet only settings: lnd autopilot lnd keysend circuibreaker lnd autounlock StaticChannelBackup to DropBox and USB * cln FUNDING fix parsing address * cln.hsmtool: add change-password and lock options * always set password A * cached peer info * fix printing cache * fix check for existing files * handle bitcoind not running * result with newline * test line break * test new line * test new line * two vars on output * #2388 improve online check (less pinging) * used cached peer status * move chache * cach file permissions * allow sudo call * fix cache * remove double scan info * add conf info to sync screen * reorder info * add space * add space * order info * internet suppress error messages * order info * fix offering Blockchain copy * fix hostname * final ready state info * lnd unlock after provision * remove debug exit * harmonize ready state * add status to lnd unlock * update lnd unlock script * edit the unlock * remove debug echo * add debug * add debug * fix if statement * debug output * switch position of source setupdata * #1126 preparing new setup with new c-lightning (#2396) * move debug script * patch patch command * make sure setup file is sourced * remove debug output * make sure lnd is put behind tor * change indent * get fresh sync progress * always set password A * cached peer info * fix printing cache * fix check for existing files * handle bitcoind not running * result with newline * test line break * test new line * test new line * two vars on output * #2388 improve online check (less pinging) * used cached peer status * move chache * cach file permissions * allow sudo call * fix cache * remove double scan info * add conf info to sync screen * reorder info * add space * add space * order info * internet suppress error messages * order info * fix offering Blockchain copy * fix hostname * final ready state info * lnd unlock after provision * remove debug exit * harmonize ready state * add status to lnd unlock * update lnd unlock script * edit the unlock * remove debug echo * add debug * add debug * fix if statement * debug output * switch position of source setupdata * lnd.unlock: fix typo * netwok.monitor.sh debug * cln-plugin.summary: fix paths * rtl: fix permission of config on copy * CASHOUT: use aliases for lnd * rtl: install correctly for paralell chains * use CHAIN in CLN and LND menu * cln: add CASHOUT option * CLOSEALL and CASHOUT: Improve labels and comments Explaining CASHOUT in the label as discussed in: #2358 * cln.install: fix tor config * cln: installthe latest master until the next release * _commands: source _aliases only if exists * network aliases: fall back to 'main' for 'chain' * new setup: keep testnet3 blocks and chainstate * new setup: improve capitalization in menu * improve help and comments * cln: install Sparko if configured, but not present * cln: add new wallet and import seed options * fix peernum * make sure that aliases get created on lnd setup * no error if aliases not yet exist * debug state * fix network alias when not set * fix syntax error * add debug error info * mute unlocking echos * add debug wait * add debug wait * make sure info is uptodate * make alias info as defaults * rename option * update sync info for no lightning * add action string * update sync info * move name dialog * wait for sync progress info * wait for syncprogress info * fix syntax * get fresh data * make sure to disable lnd * add c-lightning to debug * add setup logs to debug output * fix syntax error * add new-force wallet * try fix call hsmtool * hsm output tool * fix output * add seed-force * refactor blitz.mnemonic.py * test seed * debug info * dump object * try check * correct putput * fix syntax * check lnd for valid seed * fix gui * add Suez install script discussed in: #2366 * cln rescue file export * get correct version * add cln export gui * cln.backup.sh cln-import * correct bytesize * generate cln wallet with passwordc * fix syntax * fix syntax * mute not needed error msg * PEERING: correct message on success * cln.install-service: fix sparko check * add Suez to menu for CLN and LND needs to be installed with the bitcoin user to be able to interact with CLN related: #2366 * debug _provison.setup.sh stop bitcoind and restart with new config to avoid rpc password error disable and enable service instead of daemon-reload CLN: don't use passwordC as seedPassword * add cln.setname.sh make lnd.setname.sh work with parallel wallets * improve comments * SYSTEM: add CLNLOG and CLNCONF options * SYSTEM menu fixes * cln: add more aliases cln, clnlog, clnconf * cln: activate the backup plugin on every install * SERVICES menu: fix chantools/CLN switch * cln: load plugins from ${netprefix}cln-plugins-enabled changed the config paths to $lightning-dir/config or /networkname/config plugins are downloaded to the SDcard: /home/bitcoin/cln-plugins-available/ symlinked and loaded automatically from: /home/bitcoin/${netprefix}cln-plugins-enabled Related: #2295 * sparko: don't show logs after install * #2425 Adding experimental Blitz WebUI & API (#2426) * no password C & D when cln * add debug echos * set defaults before * #2228 wider grep to detect nvms (#2427) * cln.hsmtool: init backup with the new wallet * cln.install: fix access to raspiblitz.conf * cln-plugin.backup: fix path to backup-cli * cln: hide unhelpful warnings during setup * remove old jinja template rendering * fix lnd unlock detection * cln: look for files in .lightning dir with sudo * cln: correct lightning name in FInalDialog + typo * cln: make sure .lightning/bitcoin dir exists * FinalDialog: make the 24 words fit * cln.install.sh: create cln config if not present * Simplify localIP detection and improve compatibility (#2432) * show tail info on provision * only show lnd options when activated * fix syntax * only show main lightning impl options for RC1 * cln: always start the lightnind.service * cln: clear before showing summary * start cln on the end of provisioning * exit 0 on cln menu * press key after single actions * remove key press on cln actions * change to none * detect cln running * fix syntax * fix lightniing info * add TODO for CLN * add clnblockheight * zty with user bitcoin * check synced to chain for cln * fix increment * try scanprogress * use cln sync detection and progress * replace LNTYPE * next line * fix spaces * fix spaces * Update README.md (#2456) Fix 404 * Fix FAQ links (#2441) * Fix invalid URL ( (#2440) * support channels (#2382) * use #2370 height optimization * adjust exit codes in menu scripts * adjust password menu exit codes * adapt shutdown for cln * settings adapt to running lightning impl * fix syntax * debug info * add debug * better height * add default values * add config entry if not there yet * change default value * Added exit info for cln * make sure to load config file if available * add sparko to menu * add default for sparko * replace default sparko entry * show sparko installed or not * add more description to sparko option * RTL for clightnign in service menu * main menu item rtl * add RTL description * debug in RTL install * install sparko on recovery * update menu with cln * rework menu options Co-authored-by: openoms <oms@tuta.io> Co-authored-by: openoms <43343391+openoms@users.noreply.github.com> Co-authored-by: rek79 <rek79@users.noreply.github.com> Co-authored-by: Bitpaint <67663265+bitpaint@users.noreply.github.com> Co-authored-by: João Thallis <joaothallis@icloud.com> Co-authored-by: Peter Flock <78184669+peterflock@users.noreply.github.com> Co-authored-by: nyxnor <nyxnor@protonmail.com>
|
I final setup reboot was still needed ... but basically the Setup process is now ready for the WebUI future. |
To prepare the setup over Webbrowser in future versions, its needed to re-arrange the setup flow in that way that all setup-data (names, password, seeds, ) are gathered up-front and then are written to a setup-file and the rest of the setup runs non-interactive.
Things to keep in mind:
The text was updated successfully, but these errors were encountered: