online_delete not work, how to make online_delete work?

[passionofvc]

online_delete seen not work, I set to 1024 but the
db folder size grow bigger than i expected.
does the complete ledger range value normal?

db ->15.4 GB (16,555,249,361 バイト)

#1 my rippled.cfg

[peer_private]
0
[peers_max]
100
[node_size]
medium

[ledger_history]
**1024**

[server]
port_rpc_admin_local
port_peer
port_ws_admin_local

[port_rpc_admin_local]
port = 5005
ip = 127.0.0.1
admin = 127.0.0.1
protocol = http

[port_peer]
port = 51235
ip = 0.0.0.0
protocol = peer

[port_ws_admin_local]
port = 6006
ip = 127.0.0.1
admin = 127.0.0.1
protocol = ws

#-------------------------------------------------------------------------------

[node_size]
medium

[node_db]
type=NuDB
path=G:\ripple\db\NuDB
open_files=2000
filter_bits=12
cache_mb=256
file_size_mb=8
file_size_mult=2
**online_delete=1024**
advisory_delete=1

[database_path]
G:\ripple\db

[debug_logfile]
G:\ripple\debug.log

[sntp_servers]
time.windows.com
time.apple.com
time.nist.gov
pool.ntp.org

[ips]
r.ripple.com 51235

[validators_file]
validators.txt

[rpc_startup]
{ "command": "log_level", "severity": "warning" }

[ssl_verify]
1

G:\ripple>RippleD.exe server_info | jq  ".result.info.complete_ledgers"
Loading: "G:\ripple\rippled.cfg"
2017-Dec-19 01:09:12 HTTPClient:NFO Connecting to 127.0.0.1:5005

"35115375-35123196,35123294-35123479,35123580-35123700,35123801-35123835,35123943-35124001,35124086-35124155,35124230-35124260,35124380-35124390,35124497-35124518,35124639-35124703,35125145-35125149,35125487-35125505,35125617-35125619,35125685-35136893"

G:\ripple>RippleD.exe server_info | jq  ".result.info.complete_ledgers"
Loading: "G:\ripple\rippled.cfg"
2017-Dec-19 01:25:52 HTTPClient:NFO Connecting to 127.0.0.1:5005

"35115375-35123196,35123294-35123479,35123580-35123700,35123801-35123835,35123943-35124001,35124086-35124155,35124230-35124260,35124380-35124390,35124497-35124518,35124639-35124703,35125145-35125149,35125487-35125505,35125617-35125619,35125685-35137197"

G:\ripple>RippleD.exe server_info | jq  ".result.info.complete_ledgers"
Loading: "G:\ripple\rippled.cfg"
2017-Dec-19 08:15:17 HTTPClient:NFO Connecting to 127.0.0.1:5005

"35115375-35123196,35123294-35123479,35123580-35123700,35123801-35123835,35123943-35124001,35124086-35124155,35124230-35124260,35124380-35124390,35124497-35124518,35124639-35124703,35125145-35125149,35125487-35125505,35125617-35125619,35125685-35143679,35143803-35143818,35143919-35143932,35144134-35144535"

G:\ripple>RippleD.exe server_info | jq  ".result.info.complete_ledgers"
Loading: "G:\ripple\rippled.cfg"
2017-Dec-19 08:35:48 HTTPClient:NFO Connecting to 127.0.0.1:5005

"35115375-35123196,35123294-35123479,35123580-35123700,35123801-35123835,35123943-35124001,35124086-35124155,35124230-35124260,35124380-35124390,35124497-35124518,35124639-35124703,35125145-35125149,35125487-35125505,35125617-35125619,35125685-35143679,35143780-35144921"

G:\ripple>RippleD.exe server_info | jq  ".result.info.complete_ledgers"
Loading: "G:\ripple\rippled.cfg"
2017-Dec-19 11:13:32 HTTPClient:NFO Connecting to 127.0.0.1:5005

"35115375-35123196,35123294-35123479,35123580-35123700,35123801-35123835,35123943-35124001,35124086-35124155,35124230-35124260,35124380-35124390,35124497-35124518,35124639-35124703,35125145-35125149,35125487-35125505,35125617-35125619,35125685-35143679,35143780-35147773"

from	to	diff
35115375	35123196	7821
35123294	35123479	185
35123580	35123700	120
35123801	35123835	34
35123943	35124001	58
35124086	35124155	69
35124230	35124260	30
35124380	35124390	10
35124497	35124518	21
35124639	35124703	64
35125145	35125149	4
35125487	35125505	18
35125617	35125619	2
35125685	35143679	17994
35143780	35147773	3993

[passionofvc]

P.S:
According to
https://wiki.ripple.com/Online_deletion#Online_Deletion_Process_-_Without_advisory_delete
does it mean than must also run in manual by can_delete now,otherwise it not truncate the old ledger?

[scottschurr]

First, thank you for including the contents of the rippled.cfg file. That's a big help.

Second, it looks like you figured out the problem yourself. If you want online_delete to run automatically, then you must set advisory_delete=0.

With advisory_delete=1 your server will wait for a can_delete RPC command before it removes old ledgers. The can_delete command requires administrative privileges. Some folks want to be in control of when the delete happens. It sounds like that is not your case, so you should set advisory_delete=0.

Another choice would be to omit the advisory_delete line from your config file altogether. The setting defaults to off.

[passionofvc]

Hi @scottschurr

I run can_delete now, it shows below output,
but db size not decreased.

db size : 22.3 GB (24,022,651,110 バイト)

G:\ripple>RippleD.exe can_delete now
Loading: "G:\ripple\rippled.cfg"
2017-Dec-20 01:02:12 HTTPClient:NFO Connecting to 127.0.0.1:5005

{
   "id" : 1,
   "result" : {
      "can_delete" : 35115812,
      "status" : "success"
   }
}

the complete_ledgers

35115375 | 35123196
35123294 | 35123479
35123580 | 35123700
35123801 | 35123835
35123943 | 35124001
35124086 | 35124155
35124230 | 35124260
35124380 | 35124390
35124497 | 35124518
35124639 | 35124703
35125145 | 35125149
35125487 | 35125505
35125617 | 35125619
35125685 | 35143679
35143780 | 35148705
35148808 | 35148821
35148961 | 35149214
35149311 | 35149409
35149521 |  
35149893 |  
35150019 |  
35150156 | 35150173
35150277 | 35150414
35150652 | 35150680
35150812 |  
35150942 | 35162530

[mtrippled]

The size for sqlite database files will not decrease. However, the online_delete process made room for new data to be put into the files.

[mtrippled]

More information is here: https://sqlite.org/lang_vacuum.html
Basically, actually shrinking the files would be a time-consuming process in which rippled would not be running. Instead of this, the space is freed up within the data files so that new data will use the freed up space, and the files themselves will grow again when that space fills up.

[passionofvc]

the online_delete process made room for new data to be put into the files

if this work, the db folder will Keep size constant,
but I found nudb.dat is very large, and going bigger.

G:\ripple>dir  G:\ripple\db\NuDB\rippledb.ebe0

 Directory of G:\ripple\db\NuDB\rippledb.ebe0

2017/12/18  14:11    <DIR>          .
2017/12/18  14:11    <DIR>          ..
2017/12/20  11:51    17,424,166,499 nudb.dat
2017/12/20  11:51     2,820,116,480 nudb.key
2017/12/20  11:51           524,086 nudb.log
               3 File(s) 20,244,807,065 bytes
               2 Dir(s)  40,224,989,184 bytes free

[miguelportilla]

Hi @passionofvc ,

I would like to point out that the following settings only apply to RocksDB and are safely ignored with NuDB.

open_files=2000
filter_bits=12
cache_mb=256
file_size_mb=8
file_size_mult=2

Here is the link to the can_delete command: https://ripple.com/build/rippled-apis/#can-delete

The can_delete command takes time to process. After issuing the command, you may receive a success code meaning rotating the databases will commence accordingly but the actual process takes time. If during the process the server mode drops from full or the age of the validated ledger is over 60 seconds, the process will be interrupted and the rotation will not occur. So if the server is busy or has difficulty maintaining synchronization, the rotation operation will not succeed. If this is the case and your debug log is set to warn level, you will find the Not deleting message logged. Whether you manually issue the can_delete or have the server automatically rotate, the same rules apply.

It is worth noting that NuDB is an append-only database and unless a successful rotation occurs, the database files will continue to grow as you observed. I recommend running with a warning debug level and inspecting the log for the aforementioned warning messages.

[passionofvc]

Hi @scottschurr
I am very grateful for your reply,
I now run rippled in window server2012,
from now on I will plan to run ripple daemon in Centos7.0 for production env.
Also to use RocksDB setting, rippled running in window os not work for me,
Troubleshooting is easy to trap.

[miguelportilla]

@passionofvc RocksDB is not supported in Windows. Were you able to find any Not deleting messages in your debug log?

[b0tm1nd]

I am having the same issue.
Manually Deleted the 200GB+ database, restarted and it keeps growing again.

System: Ubuntu 20.04
rippled version 1.9.4
/var/lib/rippled and subdirectories/files ownded by rippled:rippled

Configuration excerpt:

[node_db]
online_delete=512
advisory_delete=0
[ledger_history]
512
[rpc_startup]
{ "command": "log_level", "severity": "warning" }
{ "command": "log_level", partition: "SHAMapStore", "severity": "debug" }

Regarding SHAMapStore related errors in debug.log, many of this kind:
2023-Sep-16 23:20:09.776223114 UTC SHAMapStore:WRN Waiting 5s for node to stabilize. state: connected. age 2s

Regarding delete keyword in log:

# grep -i delet  /var/log/rippled/debug.log |tail
2023-Sep-16 04:33:11.588011123 UTC LedgerConsensus:WRN Trying to thread to deleted node
2023-Sep-16 08:28:33.341854114 UTC LedgerConsensus:WRN Trying to thread to deleted node
2023-Sep-16 23:11:06.482958115 UTC Amendments:DBG Amendment 30CD365592B8EE40489BA01AE2F7555CAC9C983145871DC82A42A31CF5BAE7D9 (DeletableAccounts) is supported and will be up voted if not enabled on the ledger.

Trying to find out how can I keep database size as little.

[ximinez]

Regarding SHAMapStore related errors in debug.log, many of this kind: 2023-Sep-16 23:20:09.776223114 UTC SHAMapStore:WRN Waiting 5s for node to stabilize. state: connected. age 2s

If you're seeing a bunch of these messages (as opposed to a few every once in a while), then it means that your server is having trouble keeping up with the network under the extra load related to the deletion process. That almost always means your hardware is not powerful enough, frequently because of insufficient IOPS on your hard storage.