Deleting non existant images from Trash fails.

[n00b42]

I have some images in Trash (for quite a long time) that seem to be non existant anymore or in some dubious state, however I can neither delete them directly nor by "clearing" the trash bin.

I0206 164732.819 server/server.go:444] HTTP/1.1 POST /v2/sync/delete (UserID:819644157)
F0206 164732.824 database/files.go:90] incRefCount("A9/qYAAAAAAAAAAAAAAAAAAAAACCc", -1) failed: s.ReadDataFile: open /data/8E/jikAAAAAAAAAAAAAAAAAAAAAAAAAAANw: no such file or directory []

The frontend is then either stuck (app) or shows a proxy 502 in the webapp (probably exception on the server end).

[rthellend]

Have you tried inspect orphans and then inspect orphans --delete ? It's similar to fsck

[rthellend]

BTW, if you can find a way to reproduce this problem, please let me know.

[rthellend]

I looked at the inspect code, and I don't think it'll actually be able to delete your trash file. It'll probably just show Missing file: ... .... The output should have something that looks like a filename that ends with .ref

What you can do to manually fix the problem is:

• Run inspect users -l and find the file for your trash fileset, then
• Run inspect edit fs <that file> and manually remove the record for the broken file. It will be one that contains a string that looks like the .ref file above, but without the .ref extension.

Let me know if that solves your problem, or if you need more help.

[n00b42]

/app # inspect --db /data orphans --delete
E0206 211405.141 database/db.go:395] Missing file: BF/v06aIZbnrWQJRyEp9cvH-rp_d2MdNlcqTo-Ya12flfM ()
E0206 211405.142 database/db.go:395] Missing file: E4/5IKXjdXBgdKEkUanRwEMwQmq_6L3uHcpbyYonFoxls0 (BF/v06aIZbnrWQJRyEp9cvH-rp_d2MdNlcqTo-Ya12flfM.ref)
E0206 211405.142 database/db.go:395] Missing file: 57/VzHO0aId9hz0zYRg68kLXP3dHsBQaJd3WWnGhCnsIi8 ()
E0206 211405.142 database/db.go:395] Missing file: FC/_GnYlEfVhNj-mB8wz5fFZWedUtJ8jMu1pdmR-Xklze8 (57/VzHO0aId9hz0zYRg68kLXP3dHsBQaJd3WWnGhCnsIi8.ref)
E0206 211405.142 database/db.go:395] Missing file: 44/RNsleR0GtSClt2OaI6k-5CNmjEb8HoPkJXnRgXlTQME ()
E0206 211405.142 database/db.go:395] Missing file: BF/v1XyWP0fc5Zh-8g495XbjlVd1ma_mZBVV7KayP3X7nM (44/RNsleR0GtSClt2OaI6k-5CNmjEb8HoPkJXnRgXlTQME.ref)
E0206 211405.142 database/db.go:395] Missing file: X9/2eccCd71X-Xbt7tanG48ba2VEvXXWKSKm-LHX8f2cXX ()
E0206 211405.142 database/db.go:395] Missing file: 4E/Ti2sREC1LICXPGYI0X7s2Gw0NNy-oT470kYXFJuaOIQ (X9/2eccCd71X-Xbt7tanG48ba2VEvXXWKSKm-LHX8f2cXX.ref)
E0206 211405.142 database/db.go:395] Missing file: 8X/iupN2uhsSIWeThEzN38KFzd7bE09_om_5NS54mTNJGU ()
E0206 211405.142 database/db.go:395] Missing file: 13/E-gxTEUIseVEGoSGidCKKgZ77P6LzXTxc65XGqzQNsU (8X/iupN2uhsSIWeThEzN38KFzd7bE09_om_5NS54mTNJGU.ref)
E0206 211405.142 database/db.go:395] Missing file: XE/3jFbftIHsx1Cwd4uo1kU8suxuEWOMN3WCLdk1tqVQxY ()
E0206 211405.142 database/db.go:395] Missing file: 6E/bi9oIiq-zeXR2R5_9XTcVjZaI3ag7Zke5o4TXMaMcdY (XE/3jFbftIHsx1Cwd4uo1kU8suxuEWOMN3WCLdk1tqVQxY.ref)
E0206 211405.142 database/db.go:395] Missing file: X4/pPJQhZ6SwRhOj2wJeYZSS9whm8sQoFs-SF8KS-tKvPs ()
E0206 211405.142 database/db.go:395] Missing file: XC/rXdebGyX1X4fHKJWYhH8h5z6nC3y9YuUf6v2agzdXmY (X4/pPJQhZ6SwRhOj2wJeYZSS9whm8sQoFs-SF8KS-tKvPs.ref)
E0206 211405.142 database/db.go:395] Missing file: EE/7s7o6oXUKzQL0SBPmswBlqyxXQdSyVkZElgErJcXB4o ()
E0206 211405.143 database/db.go:395] Missing file: 74/dIfSdYhzn214g64zubdyvozZR2H__e5X1aSV54LWFu4 (EE/7s7o6oXUKzQL0SBPmswBlqyxXQdSyVkZElgErJcXB4o.ref)
E0206 211405.143 database/db.go:395] Missing file: 8E/jpe-BQcPd5doTqX1iERaf0JsWifmgtTV-IOE-nIVpj4 ()
E0206 211405.143 database/db.go:395] Missing file: 15/FYjcNOCesr_sefhn6MxviwB8lBNVI7tcgVkisatt5Vg (8E/jpe-BQcPd5doTqX1iERaf0JsWifmgtTV-IOE-nIVpj4.ref)
E0206 211405.143 database/db.go:395] Missing file: XB/q-YsyWOeCly-X62xEId_BRp3sRcyFKc0Y5eW7XTH3Xc ()
E0206 211405.143 database/db.go:395] Missing file: 63/Ywrl3CcwwPMMLYStwqnkjnNwLgKWtbL8lIFv7pkqu40 (XB/q-YsyWOeCly-X62xEId_BRp3sRcyFKc0Y5eW7XTH3Xc.ref)
E0206 211405.143 database/db.go:395] Missing file: E8/6ICHh_eXquhIs6kze6lJuzWnoveKEqNZQGmPybRs0YQ ()
E0206 211405.143 database/db.go:395] Missing file: 36/NhHX9IVX7mhYYuGmK-MO2Ls37zJYlBYigQ1YsXoX_po (E8/6ICHh_eXquhIs6kze6lJuzWnoveKEqNZQGmPybRs0YQ.ref)
E0206 211405.143 database/db.go:395] Missing file: 2B/K-Xphvj0bRk9IO1pPLrE8xvncj3QiiyP0ccenEQovJ8 ()
E0206 211405.143 database/db.go:395] Missing file: 4F/T-ua-y-MlsVad-KjLgGXebT6i6j_xE1Uh_MXdnQfuSI (2B/K-Xphvj0bRk9IO1pPLrE8xvncj3QiiyP0ccenEQovJ8.ref)
E0206 211405.143 database/db.go:395] Missing file: 0C/XJn3wRSfXEhIggLS_gSuJPwdRgzmEskvf4gXvrXShS4 ()
E0206 211405.143 database/db.go:395] Missing file: F9/-VeFyFBtO6Gs2TFJKQk579Z210BdrNvpnvB6oEJ8jIU (0C/XJn3wRSfXEhIggLS_gSuJPwdRgzmEskvf4gXvrXShS4.ref)
E0206 211405.143 database/db.go:395] Missing file: CB/y-hbSKEmW7MoQ0nyaWL-4_VISXutyjE2Oaay7dtX4Hc ()
E0206 211405.143 database/db.go:395] Missing file: 48/SJiHjs_kO2YJUpr1eM4HyXC0Pamyo_9roxek59JoqrE (CB/y-hbSKEmW7MoQ0nyaWL-4_VISXutyjE2Oaay7dtX4Hc.ref)
E0206 211405.143 database/db.go:395] Missing file: 73/c6o6JNSFe36t5KZtEPqtBwmK46wsYXdr1z8PamyE1hc ()
E0206 211405.143 database/db.go:395] Missing file: 06/Bh8a94PmxnmBvX2tgdFXbo4QdovORW0PejFp4gqqXUE (73/c6o6JNSFe36t5KZtEPqtBwmK46wsYXdr1z8PamyE1hc.ref)
E0206 211405.143 database/db.go:395] Missing file: 58/WJUBINBlMOwRM90VrMpTyCz4E4kggXhCZxFbNvYMQww ()
E0206 211405.143 database/db.go:395] Missing file: 41/QX-XhiX_z3RtEXX_NSxakpirEXtmM1G-PvU16v6FXow (58/WJUBINBlMOwRM90VrMpTyCz4E4kggXhCZxFbNvYMQww.ref)
E0206 211405.143 database/db.go:395] Missing file: X9/qYOZSEkk6kNvCIvOamUw1siIELBdJG1g9BS2JzgSCCc ()
E0206 211405.143 database/db.go:395] Missing file: 8E/jikGf046h9nyXoTlwYHiuQ5cqrYs6FQYoIIWQcf2KNw (X9/qYOZSEkk6kNvCIvOamUw1siIELBdJG1g9BS2JzgSCCc.ref)
E0206 211405.144 database/db.go:395] Missing file: X2/0u8JqYpT-odlpOKN8gv8yIR4YuCE_cC4wYZ8MIN6XMs ()
E0206 211405.144 database/db.go:395] Missing file: 1F/H_JsffLvWXb_sqX5mZvz9hGpI5z2m5QwkWq7uaTIKUg (X2/0u8JqYpT-odlpOKN8gv8yIR4YuCE_cC4wYZ8MIN6XMs.ref)
E0206 211405.144 database/db.go:395] Missing file: 9B/m2KNxKB_-1QMqamvkN87tf77emk5XvS8IwyHuIbshTk ()
E0206 211405.144 database/db.go:395] Missing file: FX/-r28vPFKG2Y1PuRimrVYiljNetQZgIqzJFHU_67X_Fk (9B/m2KNxKB_-1QMqamvkN87tf77emk5XvS8IwyHuIbshTk.ref)
E0206 211405.144 database/db.go:395] Missing file: 2F/L9K-KIf-1y75TiGNP7bVCzu60yih_-XHzZ1WT8LR3dX ()
E0206 211405.144 database/db.go:395] Missing file: F7/94eL712tcWxQq4qGiXM2739RfkV2xhTCGhu3gHosyHs (2F/L9K-KIf-1y75TiGNP7bVCzu60yih_-XHzZ1WT8LR3dX.ref)
E0206 211405.144 database/db.go:395] Missing file: 8F/j_RrBYtuU78qbKXRMx59i2XVsTJabkXByEIk6_yHZvY ()
E0206 211405.144 database/db.go:395] Missing file: 82/gjas0s3r4HdtxaL8n85U7XQItsi5fg2PjsVFQSXcxhX (8F/j_RrBYtuU78qbKXRMx59i2XVsTJabkXByEIk6_yHZvY.ref)
E0206 211405.144 database/db.go:395] Missing file: 10/EF-uuCRhJpb8aOqicW60hPIhXZSBv-45kXcZpQP6LxX ()
E0206 211405.144 database/db.go:395] Missing file: 19/GR0iOx_c8fU2FRgP0LPrFeE0ER6Qcx_8TqG5zayoyPo (10/EF-uuCRhJpb8aOqicW60hPIhXZSBv-45kXcZpQP6LxX.ref)
E0206 211405.144 database/db.go:395] Missing file: 5B/W8XXpouOKvixkXhPvcBN2xsjO694bOj4qqd5iNcvy8Q ()
E0206 211405.145 database/db.go:395] Missing file: F0/8HW8ivZcCj8H-XwJhd_vne5XQQiB8PGjeRbQ8QB_QvQ (5B/W8XXpouOKvixkXhPvcBN2xsjO694bOj4qqd5iNcvy8Q.ref)
E0206 211405.145 database/db.go:395] Missing file: FX/_QwPLMgT8S83PxYuZk6LtoNXwFahOPW-OVrhNr1TkI8 ()
E0206 211405.145 database/db.go:395] Missing file: 00/XGQXYcBX4r29VXmiKOBfIQwLUfXck40ReVd-Pcwt6l4 (FX/_QwPLMgT8S83PxYuZk6LtoNXwFahOPW-OVrhNr1TkI8.ref)
E0206 211405.145 database/db.go:395] Missing file: FB/-xIH9IeMQmXSF1p0RP0Bxy6pzN_z5aIYT9ZKd2LzgZ8 ()
E0206 211405.145 database/db.go:395] Missing file: 13/ExQb05ma_N3ezhVHQF7XNvXP9o_C1wBRnXY2S2tzpJg (FB/-xIH9IeMQmXSF1p0RP0Bxy6pzN_z5aIYT9ZKd2LzgZ8.ref)
E0206 211405.145 database/db.go:395] Missing file: 36/NpobGTyvNW10YP-bhXBfSkwxY3XNZKeOTXgmlLC70aY ()
E0206 211405.145 database/db.go:395] Missing file: XE/3g2jXYvq9PvX87S4gNr8YLUFfa1t9LaGTs3Bl70zwno (36/NpobGTyvNW10YP-bhXBfSkwxY3XNZKeOTXgmlLC70aY.ref)
E0206 211405.145 database/db.go:395] Missing file: XF/ry2pBr622MkJhQQ6UvSfKuvp-b0OhopcFqfLBb8Il94 ()
E0206 211405.145 database/db.go:395] Missing file: F5/9ScwlnrVNPoJSbMrT6Zwnic3wa9Xh9XlGbvgmhpGBzX (XF/ry2pBr622MkJhQQ6UvSfKuvp-b0OhopcFqfLBb8Il94.ref)
E0206 211405.145 database/db.go:395] Missing file: 53/U2tq5OpKfaWShY6QHRRCqcW19oL2u9R8I8bd4B3lzzo ()
E0206 211405.145 database/db.go:395] Missing file: 2X/LS2gFEbKiTbqc2m9cKvk5C-6XZypQI07KvXsuPBip84 (53/U2tq5OpKfaWShY6QHRRCqcW19oL2u9R8I8bd4B3lzzo.ref)
E0206 211405.146 database/db.go:395] Missing file: 4X/SqzyFHXodjFu9Xu_e2JHNYRtIOOvHOVa3XPcuFhEhJX ()
E0206 211405.146 database/db.go:395] Missing file: 5B/Wz1nuyGyOt7GX9z6ka2aVxyS19MJ2piFSGi2RBUex3I (4X/SqzyFHXodjFu9Xu_e2JHNYRtIOOvHOVa3XPcuFhEhJX.ref)
E0206 211405.146 database/db.go:395] Missing file: 32/MjPULsr4vMcty15hrJyt5UTVMEQfKkmgnK-8r9nfP6g ()
E0206 211405.146 database/db.go:395] Missing file: BX/vY-uofo7TzycUnjuuSNlFeQBnUBX-O_bIlvmSz95QYX (32/MjPULsr4vMcty15hrJyt5UTVMEQfKkmgnK-8r9nfP6g.ref)
E0206 211405.146 database/db.go:395] Missing file: XE/3lQpXqGqXvT5E_46W59UNvcLKan5zZVYwwagMHXpK6g ()
E0206 211405.146 database/db.go:395] Missing file: XF/3xaqQ-yFStomQXGwNzUzP_tXt9qYH9x8Et08juiMLq8 (XE/3lQpXqGqXvT5E_46W59UNvcLKan5zZVYwwagMHXpK6g.ref)
E0206 211405.146 database/db.go:395] Missing file: EF/7wvtUyXCJug5CMONz1FpLEjM75wZ1pOr9LFtxPwCOOU ()
E0206 211405.146 database/db.go:395] Missing file: X7/17d8JcVXoGJ5ihbKUOu5GF6XXZu_rBddoqVOWsqHoso (EF/7wvtUyXCJug5CMONz1FpLEjM75wZ1pOr9LFtxPwCOOU.ref)
E0206 211405.146 database/db.go:395] Missing file: X7/p71KJ1XTXVb0JHCh1X4mpPHU-LnIjUwKRzqo0RKkIdY ()
E0206 211405.146 database/db.go:395] Missing file: 6X/av4MgM0PRslrkU5Wkd3X6Xq29YRMI8VVsQ4sXsOxS6Q (X7/p71KJ1XTXVb0JHCh1X4mpPHU-LnIjUwKRzqo0RKkIdY.ref)
E0206 211405.146 database/db.go:395] Missing file: E6/5hHkuzUkglyaF0GQLh2f66O_Ly_GYXL7OZB4ScqjX-o ()
E0206 211405.146 database/db.go:395] Missing file: X8/2MZeOJW2srscsgwSest453bON1052TtZCVQyh4bB4Nk (E6/5hHkuzUkglyaF0GQLh2f66O_Ly_GYXL7OZB4ScqjX-o.ref)
E0206 211405.146 database/db.go:395] Missing file: 62/YpbICbPXhoQlesnWGHjihvWJbZp2z2Bvr-J32vOzMKQ ()
E0206 211405.146 database/db.go:395] Missing file: E3/4_yXXEaGoj64uhQ-h_3JBjcNIjzapju9nSR_XuH0dYo (62/YpbICbPXhoQlesnWGHjihvWJbZp2z2Bvr-J32vOzMKQ.ref)
E0206 211405.146 database/db.go:395] Missing file: 1C/HGEJlB8r6N_mYXUTWe3_jJbebry6CLp0dnShvX7kqYo ()
E0206 211405.146 database/db.go:395] Missing file: 3C/PXiLuzG0dq4EXSKXL1QrdlIN9_z324CqqekMLJWrtog (1C/HGEJlB8r6N_mYXUTWe3_jJbebry6CLp0dnShvX7kqYo.ref)
E0206 211405.146 database/db.go:395] Missing file: 70/cHyRWn8SPNpKZLWkRVRw9iXPlvX1uXHLuci5yL0lfag ()
E0206 211405.146 database/db.go:395] Missing file: X9/qVMF7XExSliSKSS08U-U02byz27VVYRhedlhX8xdQg4 (70/cHyRWn8SPNpKZLWkRVRw9iXPlvX1uXHLuci5yL0lfag.ref)
E0206 211405.146 database/db.go:395] Missing file: 67/Z8qmSChRNSvhcnkj5ba_yeC1owNf2S7U5J4-fjN1kcY ()
E0206 211405.146 database/db.go:395] Missing file: CB/y6G08tXwNS324K2jkEcaO4XKVCeaUMjTesOCsd_yH1o (67/Z8qmSChRNSvhcnkj5ba_yeC1owNf2S7U5J4-fjN1kcY.ref)
E0206 211405.146 database/db.go:395] Missing file: 2X/KuayMwb239YBzjXCnXedB9jBwEQMGIXi6ZK1RkGEtF0 ()
E0206 211405.146 database/db.go:395] Missing file: 20/IHP_v0ogPfKn5PUQGsGxhclGXEQiMMrZOVQxnM9u1F0 (2X/KuayMwb239YBzjXCnXedB9jBwEQMGIXi6ZK1RkGEtF0.ref)
E0206 211405.146 database/db.go:395] Missing file: 12/EoS-bXHIfBKnwEP7VI6I8drbklxkN-2vicQ5fbLCdTM ()
E0206 211405.146 database/db.go:395] Missing file: 78/eGK75zYWuYc9wQzJzEblY4XwmwKw9rqXQYZvBokr5Es (12/EoS-bXHIfBKnwEP7VI6I8drbklxkN-2vicQ5fbLCdTM.ref)
I0206 211405.163 database/db.go:405] Deleting orphan file: 0X/XV9fXaeEzh6vw0Ch7lXuvFH80eoxKuKOBucwnfCuXLo

Repeated calls do not change the output. (only the ordering of the Missing files

[rthellend]

I'll update inspect orphans to remove broken file references. Give me a few hours. I'll let you know when it's ready.

[rthellend]

I commited a change to the inspect command and updated c2fmzq/c2fmzq-server:n00b42

Before you do anything, I suggest making sure you have a current backup of your data. That's always a good idea.

Now, when you run inspect orphans, it should list any broken file references. When you add --delete, it will actually remove all the file records with such broken references.

[n00b42]

Hi, sry, I already did the manual cleanup using inspect edit fs as you suggested.
However, the clients (app and web) still showed them, but after the cleanup I was eventually able to delete them in the clients.

Just to be sure, I updated the docker image and run inspect orphans --delete to at least provide feedback that nothing bad seems to have happened so far.

[rthellend]

OK. If you have that problem again, running inspect orphans --delete should clean up the data in the clients too.

[n00b42]

ok, thank you.