filesystem corruption #107

jlpoolen · 2021-02-11T01:53:45Z

I had my moonfire-nvr running along for months with an 8 TB cap on four video feeds. On a raspberry Pi4. When I halted it and tried to start up, the process would commence and then die. I therefore tried "moonfire-nvr check" to see if that would rectify any database issue and there were several lines printed out. I then tried "moonfire-nvr upgrade" to see if that might fix the database issues.

I commenced the upgrade near 1:00 p.m. and it is over 4 hours later. In a separate console, I've been monitoring the database files and see that they are being updated.

Here's the console where I launched the update:

root@raspberrypi:/home/jlpoole# moonfire-nvr upgrade
I0210 122631.030 main moonfire_db::upgrade] Upgrading database from version 5 to version 5...
I0210 122631.051 main moonfire_db::upgrade] ...database now in journal_mode delete (requested delete).
I0210 122631.066 main moonfire_db::upgrade] ...database now in journal_mode wal (requested wal).
I0210 122631.066 main moonfire_db::upgrade] ...vacuuming database after upgrade.

Here's the other console showing listing of the the SQLite database files:

jlpoole@raspberrypi:~ $ sudo ls -la /var/lib/moonfire-nvr/db
total 1155968
drwx------ 2 moonfire-nvr moonfire-nvr 4096 Feb 10 12:26 .
drwxr-xr-x 5 moonfire-nvr moonfire-nvr 4096 Jul 10 2020 ..
-rw-r--r-- 1 moonfire-nvr moonfire-nvr 1022054400 Feb 10 12:26 db
-rw-r--r-- 1 moonfire-nvr moonfire-nvr 327680 Feb 10 14:25 db-shm
-rw-r--r-- 1 moonfire-nvr moonfire-nvr 161314512 Feb 10 14:25 db-wal
jlpoole@raspberrypi:~ $ sudo ls -lah /var/lib/moonfire-nvr/db
total 1.2G
drwx------ 2 moonfire-nvr moonfire-nvr 4.0K Feb 10 12:26 .
drwxr-xr-x 5 moonfire-nvr moonfire-nvr 4.0K Jul 10 2020 ..
-rw-r--r-- 1 moonfire-nvr moonfire-nvr 975M Feb 10 12:26 db
-rw-r--r-- 1 moonfire-nvr moonfire-nvr 320K Feb 10 14:25 db-shm
-rw-r--r-- 1 moonfire-nvr moonfire-nvr 155M Feb 10 14:25 db-wal
jlpoole@raspberrypi:~ $ sudo ls -lah /var/lib/moonfire-nvr/db
total 1.2G
drwx------ 2 moonfire-nvr moonfire-nvr 4.0K Feb 10 12:26 .
drwxr-xr-x 5 moonfire-nvr moonfire-nvr 4.0K Jul 10 2020 ..
-rw-r--r-- 1 moonfire-nvr moonfire-nvr 975M Feb 10 12:26 db
-rw-r--r-- 1 moonfire-nvr moonfire-nvr 320K Feb 10 14:26 db-shm
-rw-r--r-- 1 moonfire-nvr moonfire-nvr 155M Feb 10 14:26 db-wal
jlpoole@raspberrypi:~ $ sudo ls -lah /var/lib/moonfire-nvr/db
[sudo] password for jlpoole:
total 1.2G
drwx------ 2 moonfire-nvr moonfire-nvr 4.0K Feb 10 12:26 .
drwxr-xr-x 5 moonfire-nvr moonfire-nvr 4.0K Jul 10 2020 ..
-rw-r--r-- 1 moonfire-nvr moonfire-nvr 975M Feb 10 12:26 db
-rw-r--r-- 1 moonfire-nvr moonfire-nvr 352K Feb 10 14:47 db-shm
-rw-r--r-- 1 moonfire-nvr moonfire-nvr 166M Feb 10 14:47 db-wal
jlpoole@raspberrypi:~ $ sudo ls -lah /var/lib/moonfire-nvr/db
total 1.2G
drwx------ 2 moonfire-nvr moonfire-nvr 4.0K Feb 10 12:26 .
drwxr-xr-x 5 moonfire-nvr moonfire-nvr 4.0K Jul 10 2020 ..
-rw-r--r-- 1 moonfire-nvr moonfire-nvr 975M Feb 10 12:26 db
-rw-r--r-- 1 moonfire-nvr moonfire-nvr 352K Feb 10 14:59 db-shm
-rw-r--r-- 1 moonfire-nvr moonfire-nvr 172M Feb 10 14:59 db-wal
jlpoole@raspberrypi:~ $ ps -efww |grep moonfire
root 18553 18539 0 12:26 pts/3 00:00:38 moonfire-nvr upgrade
jlpoole 18896 18772 0 15:01 pts/5 00:00:00 grep --color=auto moonfire
jlpoole@raspberrypi:~ $ sudo ls -lah /var/lib/moonfire-nvr/db
[sudo] password for jlpoole:
total 1.2G
drwx------ 2 moonfire-nvr moonfire-nvr 4.0K Feb 10 12:26 .
drwxr-xr-x 5 moonfire-nvr moonfire-nvr 4.0K Jul 10 2020 ..
-rw-r--r-- 1 moonfire-nvr moonfire-nvr 975M Feb 10 12:26 db
-rw-r--r-- 1 moonfire-nvr moonfire-nvr 384K Feb 10 15:26 db-shm
-rw-r--r-- 1 moonfire-nvr moonfire-nvr 182M Feb 10 15:26 db-wal
jlpoole@raspberrypi:~ $ sudo ls -lah /var/lib/moonfire-nvr/db
[sudo] password for jlpoole:
total 1.2G
drwx------ 2 moonfire-nvr moonfire-nvr 4.0K Feb 10 12:26 .
drwxr-xr-x 5 moonfire-nvr moonfire-nvr 4.0K Jul 10 2020 ..
-rw-r--r-- 1 moonfire-nvr moonfire-nvr 975M Feb 10 12:26 db
-rw-r--r-- 1 moonfire-nvr moonfire-nvr 416K Feb 10 17:38 db-shm
-rw-r--r-- 1 moonfire-nvr moonfire-nvr 210M Feb 10 17:38 db-wal
jlpoole@raspberrypi:~ $ sudo ls -lah /var/lib/moonfire-nvr/db
total 1.2G
drwx------ 2 moonfire-nvr moonfire-nvr 4.0K Feb 10 12:26 .
drwxr-xr-x 5 moonfire-nvr moonfire-nvr 4.0K Jul 10 2020 ..
-rw-r--r-- 1 moonfire-nvr moonfire-nvr 975M Feb 10 12:26 db
-rw-r--r-- 1 moonfire-nvr moonfire-nvr 416K Feb 10 17:39 db-shm
-rw-r--r-- 1 moonfire-nvr moonfire-nvr 210M Feb 10 17:38 db-wal
jlpoole@raspberrypi:~ $

Question: for 8 TBs worth of video (1920x1200), should a database upgrade take over 4 hours? Is there a metric I can use to predict when the upgrade will finish? Or am I in doldrums and should kill the process? While I wait for this to resolve, of course, my ability to record is off-line, so knowing what an "upgrade" should cost in terms of time is important as one may want to weigh the risk of downtime.

scottlamb · 2021-02-11T04:59:02Z

When I halted it and tried to start up, the process would commence and then die. I therefore tried "moonfire-nvr check" to see if that would rectify any database issue and there were several lines printed out. I then tried "moonfire-nvr upgrade" to see if that might fix the database issues.

Unless the error was "Database schema version {} is too old (expected {}); see upgrade instructions in guide/upgrade.md., I doubt the upgrade will help. :-( I'm happy to look into the problem if you open another issue for the error (and that moonfire-nvr check output might be helpful).

Question: for 8 TBs worth of video (1920x1200), should a database upgrade take over 4 hours? Is there a metric I can use to predict when the upgrade will finish? Or am I in doldrums and should kill the process? While I wait for this to resolve, of course, my ability to record is off-line, so knowing what an "upgrade" should cost in terms of time is important as one may want to weigh the risk of downtime.

Wow, that is slow! But if it's still writing, I'd expect it to eventually complete (given unlimited patience...). But it might not be until your db-wal is about as big as your db; if so we're talking like 19 hours in total. Is your machine swapping or something? or maybe your flash card is quite slow.

I haven't tried a vacuum on a Pi4 with a database as large as that. I'll try one shortly.

The step that it's doing is a SQLite3 vacuum. It's not strictly necessary; you can use the --no-vacuum argument to skip it. You can also ctrl-C out of it without any data loss. The upgrade procedure vacuums at the end mostly to reclaim space. If you don't care how much space it uses, the vacuum isn't necessary. Also, if you have a faster machine available, you could copy the database over there to vacuum and then back.

My primary setup right now is an Intel NUC, with a 5.1G database (vs your 975M). A vacuum there takes about 10 minutes. Admittedly, it's a much faster machine with a Samsung M.2 SSD rather than an SD card.

[slamb@nuc ~]$ sudo -u moonfire-nvr time sqlite3 /var/lib/moonfire-nvr/db/db vacuum
42.48user 95.25system 10:51.05elapsed 21%CPU (0avgtext+0avgdata 9460maxresident)k
37578224inputs+32006840outputs (10major+1691minor)pagefaults 0swaps
[slamb@nuc ~]$ sudo ls -laFh /var/lib/moonfire-nvr/db/db
-rw------- 1 moonfire-nvr nogroup 5.1G Feb 10 20:19 /var/lib/moonfire-nvr/db/db

btw, in looking into this I discovered a minor bug. My code tries to set the page_size (to reduce the number of IO operations required for dealing with the recording_playback table in particular which has large rows):

      // WAL is the preferred journal mode for normal operation; it reduces the number of syncs
      // without compromising safety.
      set_journal_mode(&conn, "wal")?;
      if !args.no_vacuum {
          info!("...vacuuming database after upgrade.");
          conn.execute_batch(r#"
              pragma page_size = 16384;
              vacuum;
          "#)?;
      }

but the SQLite3 docs for pragma page_size say this:

Specifying a new page size does not change the page size immediately. Instead, the new page size is remembered and is used to set the page size when the database is first created, if it does not already exist when the page_size pragma is issued, or at the next VACUUM command that is run on the same database connection while not in WAL mode.

I'd intended for that pragma page_size to take effect with that vacuum, but apparently a vacuum while in WAL mode doesn't honor that. I'll move it before the set_journal_mode so it actually works...

scottlamb · 2021-02-11T05:50:27Z

What does sqlite3 --version print for you? I also realize doing the vacuum after switching back to WAL mode is not ideal for old versions of SQLite3, as mentioned here:

WAL works best with smaller transactions. WAL does not work well for very large transactions. For transactions larger than about 100 megabytes, traditional rollback journal modes will likely be faster. For transactions in excess of a gigabyte, WAL mode may fail with an I/O or disk-full error. It is recommended that one of the rollback journal modes be used for transactions larger than a few dozen megabytes. Beginning with version 3.11.0 (2016-02-15), WAL mode works as efficiently with large transactions as does rollback mode.

scottlamb · 2021-02-11T06:16:07Z

Actually, my Pi4 does have a comparably sized database, and it's much faster there. Best case: you're using the old SQLite version which is why it's slow for you. Worst case: your microSD card is failing. :-/

[slamb@pi4 ~/db-copy]$ ls -laFh
total 929M
drwx------  2 slamb slamb 4.0K Feb 10 22:04 ./
drwxr-xr-x 22 slamb slamb 4.0K Feb 10 22:04 ../
-rw-------  1 slamb slamb 929M Feb 10 21:39 db
[slamb@pi4 ~/db-copy]$ sqlite3 --version
3.31.1 2020-01-27 19:55:54 3bfa9cc97da10598521b342961df8f5f68c7388fa117345eeb516eaa837balt1
[slamb@pi4 ~/db-copy]$ time sqlite3 db 'pragma page_size; pragma journal_mode; vacuum'
1024
wal

real    6m8.492s
user    1m3.383s
sys     1m23.232s

* give a rule of thumb for update time in the documentation * log the SQLite3 version, which can affect performance * do the vacuum in non-WAL mode, to correctly set the page size and to avoid very slow behavior on older SQLite3 versions. Larger page sizes are generally faster (including subsequent vacuum operations). This won't help much for the first vacuum after this change, but it will help afterward. * likewise, set the page size properly on "moonfire-nvr init".

jlpoolen · 2021-02-11T15:42:13Z

Haven't fully digested or considered Scott's comments above. But I left my process running and inserted a "date" command to log when the process finished. Conclusion: ~18 hours.

Here's the console session started shortly after 12 noon on Wednesday, Feb 10th:

root@raspberrypi:/home/jlpoole# moonfire-nvr upgrade
I0210 122631.030 main moonfire_db::upgrade] Upgrading database from version 5 to version 5...
I0210 122631.051 main moonfire_db::upgrade] ...database now in journal_mode delete (requested delete).
I0210 122631.066 main moonfire_db::upgrade] ...database now in journal_mode wal (requested wal).
I0210 122631.066 main moonfire_db::upgrade] ...vacuuming database after upgrade.

date

I0211 061913.787 main moonfire_db::upgrade] ...done.
I0211 061914.322 main moonfire_nvr] Success.
root@raspberrypi:/home/jlpoole#
root@raspberrypi:/home/jlpoole# date
Thu 11 Feb 2021 06:19:14 AM PST
root@raspberrypi:/home/jlpoole#

The other console that was monitoring the file sizes of the databases:

jlpoole@raspberrypi:~ $ sudo ls -lah /var/lib/moonfire-nvr/db
[sudo] password for jlpoole:
total 937M
drwx------ 2 moonfire-nvr moonfire-nvr 4.0K Feb 11 06:19 .
drwxr-xr-x 5 moonfire-nvr moonfire-nvr 4.0K Jul 10  2020 ..
-rw-r--r-- 1 moonfire-nvr moonfire-nvr 937M Feb 11 06:19 db
jlpoole@raspberrypi:~

I then performed a check:

root@raspberrypi:/home/jlpoole# moonfire-nvr check > check_results_202102110727.log 2>&1
root@raspberrypi:/home/jlpoole# ls -lah check*
-rw-r--r-- 1 root root 342K Feb 11 07:27 check_results_202102110727.log
root@raspberrypi:/home/jlpoole#

The results are for 6 months available at: https://pastebin.com/A3wu0Mkz

Lastly, I tried to start up again and failed:

root@raspberrypi:/home/jlpoole# sudo moonfire-nvr run
I0211 073748.704 main moonfire_db::db] Loading video sample entries
I0211 073748.704 main moonfire_db::db] Loaded 2 video sample entries
I0211 073748.705 main moonfire_db::db] Loading sample file dirs
I0211 073748.705 main moonfire_db::db] Loaded 5 sample file dirs
I0211 073748.705 main moonfire_db::db] Loading cameras
I0211 073748.705 main moonfire_db::db] Loaded 4 cameras
I0211 073748.705 main moonfire_db::db] Loading streams
I0211 073748.705 main moonfire_db::db] Loaded 4 streams
I0211 073748.705 main moonfire_db::db] Loading recordings for camera garage_west stream MAIN
I0211 073749.099 main moonfire_db::db] Loaded 35722 recordings for camera garage_west stream MAIN
I0211 073749.099 main moonfire_db::db] Loading recordings for camera garage_east stream MAIN
I0211 073749.487 main moonfire_db::db] Loaded 35459 recordings for camera garage_east stream MAIN
I0211 073749.488 main moonfire_db::db] Loading recordings for camera peck_east stream MAIN
I0211 073750.022 main moonfire_db::db] Loaded 48906 recordings for camera peck_east stream MAIN
I0211 073750.022 main moonfire_db::db] Loading recordings for camera peck_west stream MAIN
I0211 073750.549 main moonfire_db::db] Loaded 48178 recordings for camera peck_west stream MAIN
W0211 073750.549 main moonfire_base::clock] database operation took PT1.845398777S!
I0211 073750.549 main moonfire_nvr::cmds::run] Database is loaded.
I0211 073750.556 main moonfire_db::db] Flush 1 (why: drop): no recording changes
E0211 073750.559 main moonfire_nvr] Sys(EROFS)
root@raspberrypi:/home/jlpoole#

I'll review/consider Scott's comments during the day and may not get back until this evening; I have to answer to a higher call.

scottlamb · 2021-02-11T18:53:23Z

E0211 073750.559 main moonfire_nvr] Sys(EROFS)

That's "read-only filesystem", as you can see on errno(3). (I'm preparing a commit now to improve error messages. These cryptic errors are getting old.)

I think if you take a look in your kernel logs (journalctl or dmesg), you'll see lots of error messages about I/O errors and/or filesystem corruption. :-( That would explain the slowness, the filesystem being read-only by surprise, and the many moonfire-nvr check errors that would be surprising to me if Moonfire NVR's files have been faithfully written and not changed by anything out of its control.

jlpoolen · 2021-02-11T18:57:52Z

Yup. dmesg -T is showing all kinds of problems. I was going to try to post the colorized version; it's pretty bad. I tried to reboot and no luck. I'll have to wait until this evening to debug and determine where things are.

I guess I'm the problem child. But there may be others... someday.

Edit [2/11/2021 19:46 PST]: colorized dmesg log at: http://salem1.mooo.com/moonfire/jlpoole_raspberry_pi_dmesg_Feb_11_2021.html

Inspired by the poor error message here: #107 (comment) * print the friendlier Display version of the error rather than Debug. Eg, "EROFS: Read-only filesystem" rather than "Sys(EROFS)". Do this everywhere: on command exit, on syncer retries, and on stream retries. * print the most immediate problem and additional lines for each cause. * print the backtrace or an advertisement for RUST_BACKTRACE=1 if it's unavailable. * also mention RUST_BACKTRACE=1 in the troubleshooting guide. * add context in various places, including pathnames. There are surely many places more it'd be helpful, but this is a start. * allow subcommands to return failure without an Error. In particular, "moonfire-nvr check" does its own error printing because it wants to print all the errors it finds. Printing "see earlier errors" with a meaningless stack trace seems like it'd just confuse. But I also want to get rid of the misleading "Success" at the end and 0 return to the OS.

scottlamb · 2021-02-11T19:17:52Z

No worries about being the problem child. Your reports are really helpful. I guess SD card failure is pretty common in Raspberry Pi setups; I'll update the hardware recommendations to suggest using the Samsung Endurance microSD card or putting a SATA SSD in a USB dock alongside the hard drive. Raspberry Pi 4 supports USB booting now.

With my latest commit, that error message is more informative:

[slamb@nuc ~]$ nvr run --db-dir=mydb/ --http-addr=0.0.0.0:8085
I0211 111025.091 main moonfire_nvr::cmds] Opening mydb/db in ReadWrite mode with SQLite version 3.33.0
I0211 111025.101 main moonfire_db::db] Loading video sample entries
I0211 111025.101 main moonfire_db::db] Loaded 0 video sample entries
I0211 111025.101 main moonfire_db::db] Loading sample file dirs
I0211 111025.101 main moonfire_db::db] Loaded 1 sample file dirs
I0211 111025.101 main moonfire_db::db] Loading cameras
I0211 111025.103 main moonfire_db::db] Loaded 1 cameras
I0211 111025.103 main moonfire_db::db] Loading streams
I0211 111025.103 main moonfire_db::db] Loaded 1 streams
I0211 111025.103 main moonfire_db::db] Loading recordings for camera driveway stream MAIN
I0211 111025.103 main moonfire_db::db] Loaded 0 recordings for camera driveway stream MAIN
I0211 111025.103 main moonfire_nvr::cmds::run] Database is loaded.
I0211 111025.106 main moonfire_db::db] Flush 1 (why: drop): no recording changes
E0211 111025.109 main moonfire_nvr] Exiting due to error: Failed to open dir /home/slamb/mymount/sample
caused by: Unable to open meta file
caused by: EROFS: Read-only file system

(set environment variable RUST_BACKTRACE=1 to see backtraces)
[slamb@nuc ~]$

jlpoolen · 2021-02-11T19:31:11Z

This has me thinking. Two thing on my mind: 1) I recall specifically a vendor limiting their warranty on hard drives if used for video -- hence an acknowledgement that video storage really taxes the use of the heads and 2) SSDs are not good for repeated read/writes... they have a life of 100,00-300,00 if I recall. When I was investigating my SNUC for my xen server, I had a short dialog with their engineers about SSDs and concluded that SSDs are not good for hard-hitting read/writes which may exceed their design, e.g. two years worth of intensive read/writes.

I do think my throwing a worst case at the set-up and software does help identify issues to consider. Stress testing.

scottlamb · 2021-02-11T19:58:26Z

I took a stab at updating the hardware recommendations to account for flash endurance (and strengthen the existing language about HDD endurance). Tweaks welcome.

fwiw, it's write cycles that matter, and Moonfire NVR tries to be as gentle as it can by only committing to the database once every few minutes (with the recommended flush_if_sec setting). But there is some limit, and I don't think cheap microSD cards generally state how many write cycles they support. Other load on the system (logging, compiles, general Linux stuff) also contributes to wearing them out.

scottlamb · 2021-02-11T22:39:04Z

Also, the write cycle count is per block. With write leveling, it's effectively the number of times you can rewrite the entire drive. Write amplification (extra data written because of journaling, minimum size of a write, etc.) eats into that, but it's not like x write cycles means that the device can only handle x database transactions before it fails. My understanding is that a modern, high-quality SSD will typically last for 10+ years with loads much heavier than what Moonfire NVR puts on it. This article seems consistent with that. I'm less confident in budget microSD cards, and they have no SMART monitoring to see when you're approaching the limit. I shouldn't have used a random Raspberry Pi starter kit microSD card in an example setup. Sorry!

I'm also going to add a short section to the troubleshooting guide about storage device and filesystem problems.

For recovering from corruption, as in #107. These should aid in restoring database integrity without throwing away the entire database. I only added the conditions that came up in #107, so far. * "Missing ... row" => --trash-orphan-sample-files * "Recording ... missing file" => --delete-orphan-rows * "bad video_index" => --trash-corrupt-rows

scottlamb · 2021-02-12T04:33:08Z

I think I've done all I can about this:

fixed the poor hardware recommendations
added a new troubleshooting guide section
improved error messages
improved moonfire-nvr check to spot more problems (notably SQLite3's builtin integrity check) and recover from at least the conditions mentioned in the log here.
fixed the page_size bug I happened to discover while looking into this.

When you get working hardware and a clean filesystem, you can either start from a fresh database or try running moonfire-nvr check --trash-orphan-sample-files --trash-corrupt-rows --delete-orphan-rows to get rid of the corrupted data. With one caveat: you'll have to do an upgrade to schema version 6 first, because I didn't backport it.

I've done the equivalent of these flags by hand when I had my own filesystem corruption; now it's a bit more approachable.

jlpoolen · 2021-02-15T21:30:16Z

We had a local ice storm and power went down; of course, my raspberry pi is not on a power management so it would nicely shutdown. That gives opportunity for a corrupted database. I copied your command line from above and got this error message. I built this version of moonfire a couple of days ago after wrestling with installing rustc and yarn:

root@raspberrypi:/usr/local/src/moonfire-nvr/target/release#  moonfire-nvr --version
moonfire-nvr 0.1.0
root@raspberrypi:/usr/local/src/moonfire-nvr/target/release#
moonfire-nvr check --trash-orphan-sample-files --trash-corrupt-rows --delete-orphan-rows
error: Found argument '--trash-orphan-sample-files' which wasn't expected, or isn't valid in this context

USAGE:
    moonfire-nvr check [FLAGS] [OPTIONS]

For more information try --help
root@raspberrypi:/usr/local/src/moonfire-nvr/target/release# moonfire-nvr check --trash-orphan-sample-files --trash-corrupt-rows --delete-orphan-rows
error: Found argument '--trash-orphan-sample-files' which wasn't expected, or isn't valid in this context

USAGE:
    moonfire-nvr check [FLAGS] [OPTIONS]

For more information try --help
root@raspberrypi:/usr/local/src/moonfire-nvr/target/release#

scottlamb · 2021-02-15T23:35:32Z

That output isn't from the latest version; maybe the version in your $PATH is old? Try ./moonfire-nvr, maybe?

jlpoolen · 2021-02-16T00:30:01Z

You are correct. I was in the wrong directory /usr/local/src/moonfire-nvr/target/release/ from the June 4, 2020, build, and the June 4 build was deployed to /usr/local/bin.

jlpoole@raspberrypi:/usr/local/src/moonfire-nvr $ which moonfire-nvr
/usr/local/bin/moonfire-nvr
jlpoole@raspberrypi:/usr/local/src/moonfire-nvr $ ls -la /usr/local/bin/moonfire-nvr
-rwxr-xr-x 1 root root 114304616 Jun  4  2020 /usr/local/bin/moonfire-nvr
jlpoole@raspberrypi:/usr/local/src/moonfire-nvr $

What tripped me up was not specifying the local file with a prefix "./" as you suggested, and the build target is in a different place: /usr/local/src/moonfire-nvr/server/target/release. I did not expect there to be two release directories after I brought current my git clone, so I wandered down the more familiar path of /usr/local/src/moonfire-nvr/target.

I upgraded the database from V5 to V6, then performed the cleansing operation you suggested. 634 rows and files were deleted. I started up the Feb 11 2021 build and the service is now running. Thank you.

scottlamb · 2021-02-16T01:39:15Z

Ahh, yeah, after I reorganized in dd66c7b, the top-level target directory is obsolete. Glad it's working now.

scottlamb added docs Documentation (tutorials, reference, design, etc): written or videos usability Usability / user interface improvements labels Feb 11, 2021

scottlamb changed the title ~~Amount Of Time To Upgrade?~~ filesystem corruption Feb 11, 2021

scottlamb closed this as completed Feb 12, 2021

scottlamb added the rust Rust backend work required label Feb 12, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

filesystem corruption #107

filesystem corruption #107

jlpoolen commented Feb 11, 2021

scottlamb commented Feb 11, 2021

scottlamb commented Feb 11, 2021

scottlamb commented Feb 11, 2021

jlpoolen commented Feb 11, 2021

scottlamb commented Feb 11, 2021

jlpoolen commented Feb 11, 2021 •

edited

scottlamb commented Feb 11, 2021

jlpoolen commented Feb 11, 2021

scottlamb commented Feb 11, 2021

scottlamb commented Feb 11, 2021

scottlamb commented Feb 12, 2021

jlpoolen commented Feb 15, 2021

scottlamb commented Feb 15, 2021

jlpoolen commented Feb 16, 2021

scottlamb commented Feb 16, 2021

filesystem corruption #107

filesystem corruption #107

Comments

jlpoolen commented Feb 11, 2021

scottlamb commented Feb 11, 2021

scottlamb commented Feb 11, 2021

scottlamb commented Feb 11, 2021

jlpoolen commented Feb 11, 2021

scottlamb commented Feb 11, 2021

jlpoolen commented Feb 11, 2021 • edited

scottlamb commented Feb 11, 2021

jlpoolen commented Feb 11, 2021

scottlamb commented Feb 11, 2021

scottlamb commented Feb 11, 2021

scottlamb commented Feb 12, 2021

jlpoolen commented Feb 15, 2021

scottlamb commented Feb 15, 2021

jlpoolen commented Feb 16, 2021

scottlamb commented Feb 16, 2021

jlpoolen commented Feb 11, 2021 •

edited