[Standby-Support][BUG] Disks in standby mode are not correctly processed/detected - Scrutiny will create a duplicate disk with empty data #157

garret · 2021-04-09T14:58:53Z

Describe the bug
I installed scrutiny through the linuxserver docker container on a raspberry pi 4 to which I have attached two disks by USB connection. When running for the first time scrutiny scrutiny-collector-metrics run command, my two hard disk are recognized. One is a hdd while another is a ssd. Thus, I have two entries on scrutiny (that is expected behaviour). However, I have noticed that after some time (like around a day) a new "empty" entry appears. I have marked it in red in the below screenshot. Do you know how to get rid of that fake entry?

Expected behavior
Scrutiny should only list my two disks. It does it but also shows another entry of a "phantom" disk.

Screenshots

The text was updated successfully, but these errors were encountered:

AnalogJ · 2021-04-24T21:01:18Z

adding the ability to delete disks is definitely on my backlog. I'll be closing this as a dupe of #69

AnalogJ · 2021-04-24T21:01:59Z

oh actually this looks like an error related to invalid disks being detected.

AnalogJ · 2021-04-24T21:02:10Z

Can you include your collector logs?

garret · 2021-04-25T07:50:31Z

@AnalogJ do you mean this? https://gist.github.com/garret/aef16aa7fafd9de73aee0eb57ec55315

ngkengwooi · 2021-08-16T22:01:07Z

I too have this exact issue, also on a Raspberry Pi 4 with the linuxserver docker image. In my case, /dev/sdb is a WD Elements hard disk that goes into sleep mode if not accessed after some time. I believe scrutiny detects the connected disk but is unable to collect the metrics because the disk is in sleep mode, hence the phantom entry showing "No Data". The disk usually wakes up when there's an attempt to access it. I wonder if scrutiny just needs to retry until the disk wakes up.

@garret does your disk also go into sleep mode?

garret · 2021-08-17T05:01:59Z

@ngkengwooi I have exactly the same situation as you describe in your message.

ngkengwooi · 2021-09-12T10:26:35Z

As a workaround while waiting for a fix, I added a cron job to wake the disk up (ls /mounted/path) a minute before midnight when the collector task is scheduled to kick in. On a fresh install, this seems to avoid the problem - no phantom disk has been detected after running it for 5 days. It does look like the problem is related to the disk entering sleep mode.

mark-monteiro · 2021-09-12T23:09:41Z

This also happened to me. The power cable on the hard drive came loose somehow while the system was powered on. While the disk was unplugged a check was run which caused the 'phantom disk' to appear.

AnalogJ · 2021-10-23T18:09:39Z

interesting. @ngkengwooi @garret can you possibly get me the smartctl -x -j ${DISK_ID} output for a disk when its sleeping? the smartctl --scan -j output would also be helpful

garret · 2021-10-25T16:32:05Z

@AnalogJ I don't know if I have done things right but this is my smartctl --scan -j:

root@bc99671e936f:/# smartctl --scan -j
{
  "json_format_version": [
    1,
    0
  ],
  "smartctl": {
    "version": [
      7,
      1
    ],
    "svn_revision": "5022",
    "platform_info": "armv7l-linux-5.10.17-v7l+",
    "build_info": "(local build)",
    "argv": [
      "smartctl",
      "--scan",
      "-j"
    ],
    "exit_status": 0
  },
  "devices": [
    {
      "name": "/dev/sda",
      "info_name": "/dev/sda [SAT]",
      "type": "sat",
      "protocol": "ATA"
    }
  ]
}
root@bc99671e936f:/#

I do not really know exactly what command should I give for smartctl -x -j ${DISK_ID}. I would appreciate some guidance for that.

In the meanwhile, just to recap again, when I open the scrutiny webinterface, this is what I get:

AnalogJ · 2021-10-26T03:43:32Z

Thanks @garret

Can you run the following commands, and attach the output files.

smartctl -x --json /dev/sda
smartctl -x --json /dev/sdb

garret · 2021-10-26T09:04:32Z

I am using portainer terminal to give such commands and it seems that the result is quite long and thus some of it might be cut. You tell me if these info are enough and need to find a way to give them through proper terminal:

smartctl -x --json /dev/sda:

    "power_cycle_min": 34,
    "power_cycle_max": 53,
    "lifetime_min": 28,
    "lifetime_max": 56,
    "op_limit_max": 70,
    "op_limit_min": 0,
    "limit_min": 0,
    "limit_max": 70
  },
  "ata_log_directory": {
    "gp_dir_version": 1,
    "smart_dir_version": 1,
    "smart_dir_multi_sector": true,
    "table": [
      {
        "address": 0,
        "name": "Log Directory",
        "read": true,
        "write": false,
        "gp_sectors": 1,
        "smart_sectors": 1
      },
      {
        "address": 1,
        "name": "Summary SMART error log",
        "read": true,
        "write": false,
        "smart_sectors": 1
      },
      {
        "address": 2,
        "name": "Comprehensive SMART error log",
        "read": true,
        "write": false,
        "smart_sectors": 1
      },
      {
        "address": 3,
        "name": "Ext. Comprehensive SMART error log",
        "read": true,
        "write": false,
        "gp_sectors": 1
      },
      {
        "address": 4,
        "name": "Device Statistics log",
        "read": true,
        "write": false,
        "gp_sectors": 8,
        "smart_sectors": 8
      },
      {
        "address": 6,
        "name": "SMART self-test log",
        "read": true,
        "write": false,
        "smart_sectors": 1
      },
      {
        "address": 7,
        "name": "Extended self-test log",
        "read": true,
        "write": false,
        "gp_sectors": 1
      },
      {
        "address": 9,
        "name": "Selective self-test log",
        "read": true,
        "write": true,
        "smart_sectors": 1
      },
      {
        "address": 16,
        "name": "NCQ Command Error log",
        "read": true,
        "write": false,
        "gp_sectors": 1
      },
      {
        "address": 17,
        "name": "SATA Phy Event Counters log",
        "read": true,
        "write": false,
        "gp_sectors": 1
      },
      {
        "address": 19,
        "name": "SATA NCQ Send and Receive log",
        "read": true,
        "write": false,
        "gp_sectors": 1
      },
      {
        "address": 48,
        "name": "IDENTIFY DEVICE data log",
        "read": true,
        "write": false,
        "gp_sectors": 9,
        "smart_sectors": 9
      },
      {
        "address": 128,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 129,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 130,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 131,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 132,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 133,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 134,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 135,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 136,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 137,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 138,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 139,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 140,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 141,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 142,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 143,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 144,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 145,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 146,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 147,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 148,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 149,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 150,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 151,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 152,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 153,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 154,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 155,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 156,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 157,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 158,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 159,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 161,
        "name": "Device vendor specific log",
        "smart_sectors": 16
      },
      {
        "address": 165,
        "name": "Device vendor specific log",
        "smart_sectors": 16
      },
      {
        "address": 206,
        "name": "Device vendor specific log",
        "smart_sectors": 16
      },
      {
        "address": 224,
        "name": "SCT Command/Status",
        "read": true,
        "write": true,
        "gp_sectors": 1,
        "smart_sectors": 1
      },
      {
        "address": 225,
        "name": "SCT Data Transfer",
        "read": true,
        "write": true,
        "gp_sectors": 1,
        "smart_sectors": 1
      }
    ]
  },
  "ata_smart_error_log": {
    "extended": {
      "revision": 1,
      "sectors": 1,
      "count": 0
    }
  },
  "ata_smart_self_test_log": {
    "extended": {
      "revision": 1,
      "sectors": 1,
      "count": 0
    }
  },
  "ata_smart_selective_self_test_log": {
    "revision": 1,
    "table": [
      {
        "lba_min": 0,
        "lba_max": 0,
        "status": {
          "value": 0,
          "string": "Not_testing"
        }
      },
      {
        "lba_min": 0,
        "lba_max": 0,
        "status": {
          "value": 0,
          "string": "Not_testing"
        }
      },
      {
        "lba_min": 0,
        "lba_max": 0,
        "status": {
          "value": 0,
          "string": "Not_testing"
        }
      },
      {
        "lba_min": 0,
        "lba_max": 0,
        "status": {
          "value": 0,
          "string": "Not_testing"
        }
      },
      {
        "lba_min": 0,
        "lba_max": 0,
        "status": {
          "value": 0,
          "string": "Not_testing"
        }
      }
    ],
    "current_read_scan": {
      "lba_min": 0,
      "lba_max": 65535,
      "status": {
        "value": 0,
        "string": "was never started"
      }
    },
    "flags": {
      "value": 0,
      "remainder_scan_enabled": false
    },
    "power_up_scan_resume_minutes": 0
  },
  "ata_sct_status": {
    "format_version": 3,
    "sct_version": 256,
    "device_state": {
      "value": 0,
      "string": "Active"
    },
    "temperature": {
      "current": 43,
      "power_cycle_min": 34,
      "power_cycle_max": 53,
      "lifetime_min": 28,
      "lifetime_max": 56,
      "op_limit_max": 70,
      "under_limit_count": 0,
      "over_limit_count": 0
    },
    "smart_status": {
      "passed": true
    }
  },
  "ata_sct_temperature_history": {
    "version": 2,
    "sampling_period_minutes": 10,
    "logging_interval_minutes": 10,
    "temperature": {
      "op_limit_min": 0,
      "op_limit_max": 70,
      "limit_min": 0,
      "limit_max": 70
    },
    "size": 128,
    "index": 67,
    "table": [
      36,
      36,
      36,
      36,
      37,
      36,
      38,
      35,
      36,
      37,
      36,
      36,
      35,
      37,
      38,
      37,
      36,
      37,
      39,
      36,
      37,
      36,
      37,
      36,
      36,
      36,
      37,
      38,
      39,
      36,
      37,
      38,
      38,
      37,
      37,
      37,
      36,
      36,
      36,
      39,
      37,
      36,
      37,
      37,
      38,
      39,
      38,
      38,
      39,
      37,
      38,
      38,
      39,
      37,
      38,
      37,
      38,
      37,
      38,
      37,
      36,
      38,
      37,
      39,
      40,
      38,
      37,
      39,
      39,
      38,
      37,
      37,
      37,
      39,
      37,
      38,
      39,
      39,
      40,
      38,
      38,
      37,
      37,
      37,
      37,
      36,
      37,
      37,
      37,
      37,
      38,
      39,
      37,
      37,
      36,
      37,
      36,
      37,
      38,
      36,
      36,
      36,
      36,
      36,
      38,
      37,
      36,
      40,
      37,
      36,
      37,
      39,
      38,
      42,
      42,
      42,
      42,
      41,
      41,
      42,
      42,
      42,
      42,
      42,
      42,
      42,
      42,
      42
    ]
  },
  "ata_sct_erc": {
    "read": {
      "enabled": false
    },
    "write": {
      "enabled": false
    }
  },
  "ata_device_statistics": {
    "pages": [
      {
        "number": 1,
        "name": "General Statistics",
        "revision": 1,
        "table": [
          {
            "offset": 8,
            "name": "Lifetime Power-On Resets",
            "size": 4,
            "value": 129,
            "flags": {
              "value": 192,
              "string": "V--- ",
              "valid": true,
              "normalized": false,
              "supports_dsn": false,
              "monitored_condition_met": false
            }
          },
          {
            "offset": 16,
            "name": "Power-on Hours",
            "size": 4,
            "value": 4996,
            "flags": {
              "value": 192,
              "string": "V--- ",
              "valid": true,
              "normalized": false,
              "supports_dsn": false,
              "monitored_condition_met": false
            }
          },
          {
            "offset": 24,
            "name": "Logical Sectors Written",
            "size": 6,
            "value": 47149989171,
            "flags": {
              "value": 192,
              "string": "V--- ",
              "valid": true,
              "normalized": false,
              "supports_dsn": false,
              "monitored_condition_met": false
            }
          },
          {
            "offset": 32,
            "name": "Number of Write Commands",
            "size": 6,
            "value": 188688815,
            "flags": {
              "value": 192,
              "string": "V--- ",
              "valid": true,
              "normalized": false,
              "supports_dsn": false,
              "monitored_condition_met": false
            }
          },
          {
            "offset": 40,
            "name": "Logical Sectors Read",
            "size": 6,
            "value": 61022521611,
            "flags": {
              "value": 192,
              "string": "V--- ",
              "valid": true,
              "normalized": false,
              "supports_dsn": false,
              "monitored_condition_met": false
            }
          },
          {
            "offset": 48,
            "name": "Number of Read Commands",
            "size": 6,
            "value": 195812971,
            "flags": {
              "value": 192,
              "string": "V--- ",
              "valid": true,
              "normalized": false,
              "supports_dsn": false,
              "monitored_condition_met": false
            }
          },
          {
            "offset": 56,
            "name": "Date and Time TimeStamp",
            "size": 6,
            "value": 3579000,
            "flags": {
              "value": 192,
              "string": "V--- ",
              "valid": true,
              "normalized": false,
              "supports_dsn": false,
              "monitored_condition_met": false
            }
          }
        ]
      },
      {
        "number": 4,
        "name": "General Errors Statistics",
        "revision": 1,
        "table": [
          {
            "offset": 8,
            "name": "Number of Reported Uncorrectable Errors",
            "size": 4,
            "value": 0,
            "flags": {
              "value": 192,
              "string": "V--- ",
              "valid": true,
              "normalized": false,
              "supports_dsn": false,
              "monitored_condition_met": false
            }
          },
          {
            "offset": 16,
            "name": "Resets Between Cmd Acceptance and Completion",
            "size": 4,
            "value": 1,
            "flags": {
              "value": 192,
              "string": "V--- ",
              "valid": true,
              "normalized": false,
              "supports_dsn": false,
              "monitored_condition_met": false
            }
          }
        ]
      },
      {
        "number": 5,
        "name": "Temperature Statistics",
        "revision": 1,
        "table": [
          {
            "offset": 8,
            "name": "Current Temperature",
            "size": 1,
            "value": 43,
            "flags": {
              "value": 192,
              "string": "V--- ",
              "valid": true,
              "normalized": false,
              "supports_dsn": false,
              "monitored_condition_met": false
            }
          },
          {
            "offset": 32,
            "name": "Highest Temperature",
            "size": 1,
            "value": 56,
            "flags": {
              "value": 192,
              "string": "V--- ",
              "valid": true,
              "normalized": false,
              "supports_dsn": false,
              "monitored_condition_met": false
            }
          },
          {
            "offset": 40,
            "name": "Lowest Temperature",
            "size": 1,
            "value": 28,
            "flags": {
              "value": 192,
              "string": "V--- ",
              "valid": true,
              "normalized": false,
              "supports_dsn": false,
              "monitored_condition_met": false
            }
          },
          {
            "offset": 88,
            "name": "Specified Maximum Operating Temperature",
            "size": 1,
            "value": 70,
            "flags": {
              "value": 192,
              "string": "V--- ",
              "valid": true,
              "normalized": false,
              "supports_dsn": false,
              "monitored_condition_met": false
            }
          }
        ]
      },
      {
        "number": 6,
        "name": "Transport Statistics",
        "revision": 1,
        "table": [
          {
            "offset": 8,
            "name": "Number of Hardware Resets",
            "size": 4,
            "value": 9,
            "flags": {
              "value": 192,
              "string": "V--- ",
              "valid": true,
              "normalized": false,
              "supports_dsn": false,
              "monitored_condition_met": false
            }
          },
          {
            "offset": 16,
            "name": "Number of ASR Events",
            "size": 4,
            "value": 0,
            "flags": {
              "value": 192,
              "string": "V--- ",
              "valid": true,
              "normalized": false,
              "supports_dsn": false,
              "monitored_condition_met": false
            }
          },
          {
            "offset": 24,
            "name": "Number of Interface CRC Errors",
            "size": 4,
            "value": 0,
            "flags": {
              "value": 192,
              "string": "V--- ",
              "valid": true,
              "normalized": false,
              "supports_dsn": false,
              "monitored_condition_met": false
            }
          }
        ]
      },
      {
        "number": 7,
        "name": "Solid State Device Statistics",
        "revision": 1,
        "table": [
          {
            "offset": 8,
            "name": "Percentage Used Endurance Indicator",
            "size": 1,
            "value": 1,
            "flags": {
              "value": 224,
              "string": "VN-- ",
              "valid": true,
              "normalized": true,
              "supports_dsn": false,
              "monitored_condition_met": false
            }
          }
        ]
      }
    ]
  },
  "sata_phy_event_counters": {
    "table": [
      {
        "id": 1,
        "name": "Command failed due to ICRC error",
        "size": 2,
        "value": 0,
        "overflow": false
      },
      {
        "id": 2,
        "name": "R_ERR response for data FIS",
        "size": 2,
        "value": 0,
        "overflow": false
      },
      {
        "id": 3,
        "name": "R_ERR response for device-to-host data FIS",
        "size": 2,
        "value": 0,
        "overflow": false
      },
      {
        "id": 4,
        "name": "R_ERR response for host-to-device data FIS",
        "size": 2,
        "value": 0,
        "overflow": false
      },
      {
        "id": 5,
        "name": "R_ERR response for non-data FIS",
        "size": 2,
        "value": 0,
        "overflow": false
      },
      {
        "id": 6,
        "name": "R_ERR response for device-to-host non-data FIS",
        "size": 2,
        "value": 0,
        "overflow": false
      },
      {
        "id": 7,
        "name": "R_ERR response for host-to-device non-data FIS",
        "size": 2,
        "value": 0,
        "overflow": false
      },
      {
        "id": 8,
        "name": "Device-to-host non-data FIS retries",
        "size": 2,
        "value": 0,
        "overflow": false
      },
      {
        "id": 9,
        "name": "Transition from drive PhyRdy to drive PhyNRdy",
        "size": 2,
        "value": 0,
        "overflow": false
      },
      {
        "id": 10,
        "name": "Device-to-host register FISes sent due to a COMRESET",
        "size": 2,
        "value": 0,
        "overflow": false
      },
      {
        "id": 11,
        "name": "CRC errors within host-to-device FIS",
        "size": 2,
        "value": 0,
        "overflow": false
      },
      {
        "id": 13,
        "name": "Non-CRC errors within host-to-device FIS",
        "size": 2,
        "value": 0,
        "overflow": false
      },
      {
        "id": 15,
        "name": "R_ERR response for host-to-device data FIS, CRC",
        "size": 2,
        "value": 0,
        "overflow": false
      },
      {
        "id": 16,
        "name": "R_ERR response for host-to-device data FIS, non-CRC",
        "size": 2,
        "value": 0,
        "overflow": false
      },
      {
        "id": 18,
        "name": "R_ERR response for host-to-device non-data FIS, CRC",
        "size": 2,
        "value": 0,
        "overflow": false
      },
      {
        "id": 19,
        "name": "R_ERR response for host-to-device non-data FIS, non-CRC",
        "size": 2,
        "value": 0,
        "overflow": false
      }
    ],
    "reset": false
  }
}
root@bc99671e936f:/#

smartctl -x --json /dev/sdb:

root@bc99671e936f:/# smartctl -x --json /dev/sdb
{
  "json_format_version": [
    1,
    0
  ],
  "smartctl": {
    "version": [
      7,
      1
    ],
    "svn_revision": "5022",
    "platform_info": "armv7l-linux-5.10.17-v7l+",
    "build_info": "(local build)",
    "argv": [
      "smartctl",
      "-x",
      "--json",
      "/dev/sdb"
    ],
    "messages": [
      {
        "string": "Smartctl open device: /dev/sdb [SAT] failed: No such device",
        "severity": "error"
      }
    ],
    "exit_status": 2
  }
}
root@bc99671e936f:/#

austozi · 2021-10-26T18:59:44Z

@AnalogJ, here's mine, same issue with /dev/sdb:

smartctl -x --json /dev/sda:

{
  "json_format_version": [
    1,
    0
  ],
  "smartctl": {
    "version": [
      7,
      1
    ],
    "svn_revision": "5022",
    "platform_info": "armv7l-linux-5.10.60-v7l+",
    "build_info": "(local build)",
    "argv": [
      "smartctl",
      "-x",
      "--json",
      "/dev/sda"
    ],
    "exit_status": 68
  },
  "device": {
    "name": "/dev/sda",
    "info_name": "/dev/sda [SAT]",
    "type": "sat",
    "protocol": "ATA"
  },
  "model_family": "Western Digital Blue",
  "model_name": "WDC WD20NPVZ-00WFZT0",
  "serial_number": "WD-WXG1E26E5ZJY",
  "wwn": {
    "naa": 5,
    "oui": 5358,
    "id": 10266587019
  },
  "firmware_version": "01.01A01",
  "user_capacity": {
    "blocks": 3907029168,
    "bytes": 2000398934016
  },
  "logical_block_size": 512,
  "physical_block_size": 4096,
  "rotation_rate": 5200,
  "in_smartctl_database": true,
  "ata_version": {
    "string": "ACS-2 (minor revision not indicated)",
    "major_value": 1022,
    "minor_value": 0
  },
  "sata_version": {
    "string": "SATA 3.0",
    "value": 62
  },
  "interface_speed": {
    "max": {
      "sata_value": 14,
      "string": "6.0 Gb/s",
      "units_per_second": 60,
      "bits_per_unit": 100000000
    },
    "current": {
      "sata_value": 2,
      "string": "3.0 Gb/s",
      "units_per_second": 30,
      "bits_per_unit": 100000000
    }
  },
  "local_time": {
    "time_t": 1635274064,
    "asctime": "Tue Oct 26 19:47:44 2021 BST"
  },
  "ata_apm": {
    "enabled": true,
    "level": 128,
    "string": "minimum power consumption without standby",
    "max_performance": false,
    "min_power": true,
    "with_standby": false
  },
  "read_lookahead": {
    "enabled": true
  },
  "write_cache": {
    "enabled": true
  },
  "ata_security": {
    "state": 33,
    "string": "Disabled, NOT FROZEN [SEC1]",
    "enabled": false,
    "frozen": false
  },
  "smart_status": {
    "passed": true
  },
  "ata_smart_data": {
    "offline_data_collection": {
      "status": {
        "value": 0,
        "string": "was never started"
      },
      "completion_seconds": 38160
    },
    "self_test": {
      "status": {
        "value": 0,
        "string": "completed without error",
        "passed": true
      },
      "polling_minutes": {
        "short": 2,
        "extended": 421,
        "conveyance": 5
      }
    },
    "capabilities": {
      "values": [
        123,
        3
      ],
      "exec_offline_immediate_supported": true,
      "offline_is_aborted_upon_new_cmd": false,
      "offline_surface_scan_supported": true,
      "self_tests_supported": true,
      "conveyance_self_test_supported": true,
      "selective_self_test_supported": true,
      "attribute_autosave_enabled": true,
      "error_logging_supported": true,
      "gp_logging_supported": true
    }
  },
  "ata_sct_capabilities": {
    "value": 28725,
    "error_recovery_control_supported": false,
    "feature_control_supported": true,
    "data_table_supported": true
  },
  "ata_smart_attributes": {
    "revision": 16,
    "table": [
      {
        "id": 1,
        "name": "Raw_Read_Error_Rate",
        "value": 200,
        "worst": 200,
        "thresh": 51,
        "when_failed": "",
        "flags": {
          "value": 47,
          "string": "POSR-K ",
          "prefailure": true,
          "updated_online": true,
          "performance": true,
          "error_rate": true,
          "event_count": false,
          "auto_keep": true
        },
        "raw": {
          "value": 6,
          "string": "6"
        }
      },
      {
        "id": 3,
        "name": "Spin_Up_Time",
        "value": 207,
        "worst": 193,
        "thresh": 21,
        "when_failed": "",
        "flags": {
          "value": 39,
          "string": "POS--K ",
          "prefailure": true,
          "updated_online": true,
          "performance": true,
          "error_rate": false,
          "event_count": false,
          "auto_keep": true
        },
        "raw": {
          "value": 4641,
          "string": "4641"
        }
      },
      {
        "id": 4,
        "name": "Start_Stop_Count",
        "value": 91,
        "worst": 91,
        "thresh": 0,
        "when_failed": "",
        "flags": {
          "value": 50,
          "string": "-O--CK ",
          "prefailure": false,
          "updated_online": true,
          "performance": false,
          "error_rate": false,
          "event_count": true,
          "auto_keep": true
        },
        "raw": {
          "value": 9926,
          "string": "9926"
        }
      },
      {
        "id": 5,
        "name": "Reallocated_Sector_Ct",
        "value": 200,
        "worst": 200,
        "thresh": 140,
        "when_failed": "",
        "flags": {
          "value": 51,
          "string": "PO--CK ",
          "prefailure": true,
          "updated_online": true,
          "performance": false,
          "error_rate": false,
          "event_count": true,
          "auto_keep": true
        },
        "raw": {
          "value": 0,
          "string": "0"
        }
      },
      {
        "id": 7,
        "name": "Seek_Error_Rate",
        "value": 200,
        "worst": 200,
        "thresh": 0,
        "when_failed": "",
        "flags": {
          "value": 46,
          "string": "-OSR-K ",
          "prefailure": false,
          "updated_online": true,
          "performance": true,
          "error_rate": true,
          "event_count": false,
          "auto_keep": true
        },
        "raw": {
          "value": 0,
          "string": "0"
        }
      },
      {
        "id": 9,
        "name": "Power_On_Hours",
        "value": 60,
        "worst": 60,
        "thresh": 0,
        "when_failed": "",
        "flags": {
          "value": 50,
          "string": "-O--CK ",
          "prefailure": false,
          "updated_online": true,
          "performance": false,
          "error_rate": false,
          "event_count": true,
          "auto_keep": true
        },
        "raw": {
          "value": 29434,
          "string": "29434"
        }
      },
      {
        "id": 10,
        "name": "Spin_Retry_Count",
        "value": 100,
        "worst": 100,
        "thresh": 0,
        "when_failed": "",
        "flags": {
          "value": 50,
          "string": "-O--CK ",
          "prefailure": false,
          "updated_online": true,
          "performance": false,
          "error_rate": false,
          "event_count": true,
          "auto_keep": true
        },
        "raw": {
          "value": 0,
          "string": "0"
        }
      },
      {
        "id": 11,
        "name": "Calibration_Retry_Count",
        "value": 100,
        "worst": 100,
        "thresh": 0,
        "when_failed": "",
        "flags": {
          "value": 50,
          "string": "-O--CK ",
          "prefailure": false,
          "updated_online": true,
          "performance": false,
          "error_rate": false,
          "event_count": true,
          "auto_keep": true
        },
        "raw": {
          "value": 0,
          "string": "0"
        }
      },
      {
        "id": 12,
        "name": "Power_Cycle_Count",
        "value": 100,
        "worst": 100,
        "thresh": 0,
        "when_failed": "",
        "flags": {
          "value": 50,
          "string": "-O--CK ",
          "prefailure": false,
          "updated_online": true,
          "performance": false,
          "error_rate": false,
          "event_count": true,
          "auto_keep": true
        },
        "raw": {
          "value": 374,
          "string": "374"
        }
      },
      {
        "id": 192,
        "name": "Power-Off_Retract_Count",
        "value": 200,
        "worst": 200,
        "thresh": 0,
        "when_failed": "",
        "flags": {
          "value": 50,
          "string": "-O--CK ",
          "prefailure": false,
          "updated_online": true,
          "performance": false,
          "error_rate": false,
          "event_count": true,
          "auto_keep": true
        },
        "raw": {
          "value": 282,
          "string": "282"
        }
      },
      {
        "id": 193,
        "name": "Load_Cycle_Count",
        "value": 1,
        "worst": 1,
        "thresh": 0,
        "when_failed": "",
        "flags": {
          "value": 50,
          "string": "-O--CK ",
          "prefailure": false,
          "updated_online": true,
          "performance": false,
          "error_rate": false,
          "event_count": true,
          "auto_keep": true
        },
        "raw": {
          "value": 1404847,
          "string": "1404847"
        }
      },
      {
        "id": 194,
        "name": "Temperature_Celsius",
        "value": 111,
        "worst": 103,
        "thresh": 0,
        "when_failed": "",
        "flags": {
          "value": 34,
          "string": "-O---K ",
          "prefailure": false,
          "updated_online": true,
          "performance": false,
          "error_rate": false,
          "event_count": false,
          "auto_keep": true
        },
        "raw": {
          "value": 41,
          "string": "41"
        }
      },
      {
        "id": 196,
        "name": "Reallocated_Event_Count",
        "value": 200,
        "worst": 200,
        "thresh": 0,
        "when_failed": "",
        "flags": {
          "value": 50,
          "string": "-O--CK ",
          "prefailure": false,
          "updated_online": true,
          "performance": false,
          "error_rate": false,
          "event_count": true,
          "auto_keep": true
        },
        "raw": {
          "value": 0,
          "string": "0"
        }
      },
      {
        "id": 197,
        "name": "Current_Pending_Sector",
        "value": 200,
        "worst": 200,
        "thresh": 0,
        "when_failed": "",
        "flags": {
          "value": 50,
          "string": "-O--CK ",
          "prefailure": false,
          "updated_online": true,
          "performance": false,
          "error_rate": false,
          "event_count": true,
          "auto_keep": true
        },
        "raw": {
          "value": 0,
          "string": "0"
        }
      },
      {
        "id": 198,
        "name": "Offline_Uncorrectable",
        "value": 100,
        "worst": 253,
        "thresh": 0,
        "when_failed": "",
        "flags": {
          "value": 48,
          "string": "----CK ",
          "prefailure": false,
          "updated_online": false,
          "performance": false,
          "error_rate": false,
          "event_count": true,
          "auto_keep": true
        },
        "raw": {
          "value": 0,
          "string": "0"
        }
      },
      {
        "id": 199,
        "name": "UDMA_CRC_Error_Count",
        "value": 200,
        "worst": 200,
        "thresh": 0,
        "when_failed": "",
        "flags": {
          "value": 50,
          "string": "-O--CK ",
          "prefailure": false,
          "updated_online": true,
          "performance": false,
          "error_rate": false,
          "event_count": true,
          "auto_keep": true
        },
        "raw": {
          "value": 0,
          "string": "0"
        }
      },
      {
        "id": 200,
        "name": "Multi_Zone_Error_Rate",
        "value": 100,
        "worst": 253,
        "thresh": 0,
        "when_failed": "",
        "flags": {
          "value": 8,
          "string": "---R-- ",
          "prefailure": false,
          "updated_online": false,
          "performance": false,
          "error_rate": true,
          "event_count": false,
          "auto_keep": false
        },
        "raw": {
          "value": 0,
          "string": "0"
        }
      }
    ]
  },
  "power_on_time": {
    "hours": 29434
  },
  "power_cycle_count": 374,
  "temperature": {
    "current": 41,
    "power_cycle_min": 23,
    "power_cycle_max": 41,
    "lifetime_min": 14,
    "lifetime_max": 50,
    "op_limit_min": 0,
    "op_limit_max": 60,
    "limit_min": -41,
    "limit_max": 85
  },
  "ata_log_directory": {
    "gp_dir_version": 1,
    "smart_dir_version": 1,
    "smart_dir_multi_sector": true,
    "table": [
      {
        "address": 0,
        "name": "Log Directory",
        "read": true,
        "write": false,
        "gp_sectors": 1,
        "smart_sectors": 1
      },
      {
        "address": 1,
        "name": "Summary SMART error log",
        "read": true,
        "write": false,
        "smart_sectors": 1
      },
      {
        "address": 2,
        "name": "Comprehensive SMART error log",
        "read": true,
        "write": false,
        "smart_sectors": 5
      },
      {
        "address": 3,
        "name": "Ext. Comprehensive SMART error log",
        "read": true,
        "write": false,
        "gp_sectors": 6
      },
      {
        "address": 6,
        "name": "SMART self-test log",
        "read": true,
        "write": false,
        "smart_sectors": 1
      },
      {
        "address": 7,
        "name": "Extended self-test log",
        "read": true,
        "write": false,
        "gp_sectors": 1
      },
      {
        "address": 9,
        "name": "Selective self-test log",
        "read": true,
        "write": true,
        "smart_sectors": 1
      },
      {
        "address": 16,
        "name": "NCQ Command Error log",
        "read": true,
        "write": false,
        "gp_sectors": 1
      },
      {
        "address": 17,
        "name": "SATA Phy Event Counters log",
        "read": true,
        "write": false,
        "gp_sectors": 1
      },
      {
        "address": 128,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 129,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 130,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 131,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 132,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 133,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 134,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 135,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 136,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 137,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 138,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 139,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 140,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 141,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 142,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 143,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 144,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 145,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 146,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 147,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 148,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 149,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 150,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 151,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 152,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 153,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 154,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 155,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 156,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 157,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 158,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 159,
        "name": "Host vendor specific log",
        "read": true,
        "write": true,
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 160,
        "name": "Device vendor specific log",
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 161,
        "name": "Device vendor specific log",
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 162,
        "name": "Device vendor specific log",
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 163,
        "name": "Device vendor specific log",
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 164,
        "name": "Device vendor specific log",
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 165,
        "name": "Device vendor specific log",
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 166,
        "name": "Device vendor specific log",
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 167,
        "name": "Device vendor specific log",
        "gp_sectors": 16,
        "smart_sectors": 16
      },
      {
        "address": 168,
        "name": "Device vendor specific log",
        "gp_sectors": 1,
        "smart_sectors": 1
      },
      {
        "address": 169,
        "name": "Device vendor specific log",
        "gp_sectors": 1,
        "smart_sectors": 1
      },
      {
        "address": 170,
        "name": "Device vendor specific log",
        "gp_sectors": 1,
        "smart_sectors": 1
      },
      {
        "address": 171,
        "name": "Device vendor specific log",
        "gp_sectors": 1,
        "smart_sectors": 1
      },
      {
        "address": 172,
        "name": "Device vendor specific log",
        "gp_sectors": 1,
        "smart_sectors": 1
      },
      {
        "address": 173,
        "name": "Device vendor specific log",
        "gp_sectors": 1,
        "smart_sectors": 1
      },
      {
        "address": 174,
        "name": "Device vendor specific log",
        "gp_sectors": 1,
        "smart_sectors": 1
      },
      {
        "address": 175,
        "name": "Device vendor specific log",
        "gp_sectors": 1,
        "smart_sectors": 1
      },
      {
        "address": 176,
        "name": "Device vendor specific log",
        "gp_sectors": 1,
        "smart_sectors": 1
      },
      {
        "address": 177,
        "name": "Device vendor specific log",
        "gp_sectors": 1,
        "smart_sectors": 1
      },
      {
        "address": 178,
        "name": "Device vendor specific log",
        "gp_sectors": 1,
        "smart_sectors": 1
      },
      {
        "address": 179,
        "name": "Device vendor specific log",
        "gp_sectors": 1,
        "smart_sectors": 1
      },
      {
        "address": 180,
        "name": "Device vendor specific log",
        "gp_sectors": 1,
        "smart_sectors": 1
      },
      {
        "address": 181,
        "name": "Device vendor specific log",
        "gp_sectors": 1,
        "smart_sectors": 1
      },
      {
        "address": 182,
        "name": "Device vendor specific log",
        "gp_sectors": 1,
        "smart_sectors": 1
      },
      {
        "address": 183,
        "name": "Device vendor specific log",
        "gp_sectors": 39,
        "smart_sectors": 39
      },
      {
        "address": 189,
        "name": "Device vendor specific log",
        "gp_sectors": 1,
        "smart_sectors": 1
      },
      {
        "address": 192,
        "name": "Device vendor specific log",
        "gp_sectors": 1,
        "smart_sectors": 1
      },
      {
        "address": 193,
        "name": "Device vendor specific log",
        "gp_sectors": 93
      },
      {
        "address": 224,
        "name": "SCT Command/Status",
        "read": true,
        "write": true,
        "gp_sectors": 1,
        "smart_sectors": 1
      },
      {
        "address": 225,
        "name": "SCT Data Transfer",
        "read": true,
        "write": true,
        "gp_sectors": 1,
        "smart_sectors": 1
      }
    ]
  },
  "ata_smart_error_log": {
    "extended": {
      "revision": 1,
      "sectors": 6,
      "count": 6,
      "logged_count": 6,
      "table": [
        {
          "error_number": 6,
          "log_index": 5,
          "lifetime_hours": 4811,
          "device_state": {
            "value": 3,
            "string": "active or idle"
          },
          "completion_registers": {
            "error": 64,
            "status": 81,
            "count": 8,
            "lba": 752587896,
            "device": 64,
            "device_control": 0
          },
          "error_description": "Error: UNC 8 sectors at LBA = 0x2cdb9478 = 752587896",
          "previous_commands": [
            {
              "registers": {
                "command": 37,
                "features": 0,
                "count": 8,
                "lba": 752587896,
                "device": 64,
                "device_control": 0
              },
              "powerup_milliseconds": 3430534363,
              "command_name": "READ DMA EXT"
            },
            {
              "registers": {
                "command": 37,
                "features": 0,
                "count": 8,
                "lba": 752587888,
                "device": 64,
                "device_control": 0
              },
              "powerup_milliseconds": 3430534362,
              "command_name": "READ DMA EXT"
            },
            {
              "registers": {
                "command": 37,
                "features": 0,
                "count": 8,
                "lba": 752587880,
                "device": 64,
                "device_control": 0
              },
              "powerup_milliseconds": 3430534362,
              "command_name": "READ DMA EXT"
            },
            {
              "registers": {
                "command": 37,
                "features": 0,
                "count": 8,
                "lba": 752587872,
                "device": 64,
                "device_control": 0
              },
              "powerup_milliseconds": 3430534361,
              "command_name": "READ DMA EXT"
            },
            {
              "registers": {
                "command": 37,
                "features": 0,
                "count": 8,
                "lba": 752587864,
                "device": 64,
                "device_control": 0
              },
              "powerup_milliseconds": 3430534360,
              "command_name": "READ DMA EXT"
            }
          ]
        },
        {
          "error_number": 5,
          "log_index": 4,
          "lifetime_hours": 4811,
          "device_state": {
            "value": 3,
            "string": "active or idle"
          },
          "completion_registers": {
            "error": 64,
            "status": 81,
            "count": 16,
            "lba": 752587896,
            "device": 64,
            "device_control": 0
          },
          "error_description": "Error: UNC 16 sectors at LBA = 0x2cdb9478 = 752587896",
          "previous_commands": [
            {
              "registers": {
                "command": 37,
                "features": 0,
                "count": 16,
                "lba": 752587888,
                "device": 64,
                "device_control": 0
              },
              "powerup_milliseconds": 3430530851,
              "command_name": "READ DMA EXT"
            },
            {
              "registers": {
                "command": 37,
                "features": 0,
                "count": 240,
                "lba": 752587648,
                "device": 64,
                "device_control": 0
              },
              "powerup_milliseconds": 3430530846,
              "command_name": "READ DMA EXT"
            },
            {
              "registers": {
                "command": 37,
                "features": 0,
                "count": 16,
                "lba": 752587632,
                "device": 64,
                "device_control": 0
              },
              "powerup_milliseconds": 3430530833,
              "command_name": "READ DMA EXT"
            },
            {
              "registers": {
                "command": 37,
                "features": 0,
                "count": 240,
                "lba": 752587392,
                "device": 64,
                "device_control": 0
              },
              "powerup_milliseconds": 3430530828,
              "command_name": "READ DMA EXT"
            },
            {
              "registers": {
                "command": 37,
                "features": 0,
                "count": 16,
                "lba": 752587376,
                "device": 64,
                "device_control": 0
              },
              "powerup_milliseconds": 3430530815,
              "command_name": "READ DMA EXT"
            }
          ]
        },
        {
          "error_number": 4,
          "log_index": 3,
          "lifetime_hours": 1984,
          "device_state": {
            "value": 3,
            "string": "active or idle"
          },
          "completion_registers": {
            "error": 64,
            "status": 81,
            "count": 240,
            "lba": 752587896,
            "device": 64,
            "device_control": 0
          },
          "error_description": "Error: UNC 240 sectors at LBA = 0x2cdb9478 = 752587896",
          "previous_commands": [
            {
              "registers": {
                "command": 37,
                "features": 3,
                "count": 240,
                "lba": 752587808,
                "device": 64,
                "device_control": 0
              },
              "powerup_milliseconds": 953319,
              "command_name": "READ DMA EXT"
            },
            {
              "registers": {
                "command": 239,
                "features": 3,
                "count": 70,
                "lba": 14390304,
                "device": 0,
                "device_control": 0
              },
              "powerup_milliseconds": 953319,
              "command_name": "SET FEATURES [Set transfer mode]"
            },
            {
              "registers": {
                "command": 239,
                "features": 3,
                "count": 12,
                "lba": 14390304,
                "device": 0,
                "device_control": 0
              },
              "powerup_milliseconds": 953319,
              "command_name": "SET FEATURES [Set transfer mode]"
            },
            {
              "registers": {
                "command": 236,
                "features": 218,
                "count": 240,
                "lba": 14390304,
                "device": 0,
                "device_control": 0
              },
              "powerup_milliseconds": 953318,
              "command_name": "IDENTIFY DEVICE"
            },
            {
              "registers": {
                "command": 37,
                "features": 218,
                "count": 240,
                "lba": 752587808,
                "device": 64,
                "device_control": 0
              },
              "powerup_milliseconds": 922642,
              "command_name": "READ DMA EXT"
            }
          ]
        },
        {
          "error_number": 3,
          "log_index": 2,
          "lifetime_hours": 1984,
          "device_state": {
            "value": 3,
            "string": "active or idle"
          },
          "completion_registers": {
            "error": 64,
            "status": 81,
            "count": 240,
            "lba": 752587896,
            "device": 64,
            "device_control": 0
          },
          "error_description": "Error: UNC 240 sectors at LBA = 0x2cdb9478 = 752587896",
          "previous_commands": [
            {
              "registers": {
                "command": 37,
                "features": 218,
                "count": 240,
                "lba": 752587808,
                "device": 64,
                "device_control": 0
              },
              "powerup_milliseconds": 922642,
              "command_name": "READ DMA EXT"
            },
            {
              "registers": {
                "command": 37,
                "features": 218,
                "count": 16,
                "lba": 752587792,
                "device": 64,
                "device_control": 0
              },
              "powerup_milliseconds": 922641,
              "command_name": "READ DMA EXT"
            },
            {
              "registers": {
                "command": 37,
                "features": 218,
                "count": 240,
                "lba": 752587552,
                "device": 64,
                "device_control": 0
              },
              "powerup_milliseconds": 922638,
              "command_name": "READ DMA EXT"
            },
            {
              "registers": {
                "command": 37,
                "features": 218,
                "count": 16,
                "lba": 752587536,
                "device": 64,
                "device_control": 0
              },
              "powerup_milliseconds": 922637,
              "command_name": "READ DMA EXT"
            },
            {
              "registers": {
                "command": 37,
                "features": 218,
                "count": 240,
                "lba": 752587296,
                "device": 64,
                "device_control": 0
              },
              "powerup_milliseconds": 922634,
              "command_name": "READ DMA EXT"
            }
          ]
        },
        {
          "error_number": 2,
          "log_index": 1,
          "lifetime_hours": 1984,
          "device_state": {
            "value": 3,
            "string": "active or idle"
          },
          "completion_registers": {
            "error": 64,
            "status": 81,
            "count": 240,
            "lba": 752587896,
            "device": 64,
            "device_control": 0
          },
          "error_description": "Error: UNC 240 sectors at LBA = 0x2cdb9478 = 752587896",
          "previous_commands": [
            {
              "registers": {
                "command": 37,
                "features": 3,
                "count": 240,
                "lba": 752587776,
                "device": 64,
                "device_control": 0
              },
              "powerup_milliseconds": 3899562,
              "command_name": "READ DMA EXT"
            },
            {
              "registers": {
                "command": 239,
                "features": 3,
                "count": 70,
                "lba": 14390272,
                "device": 0,
                "device_control": 0
              },
              "powerup_milliseconds": 3899562,
              "command_name": "SET FEATURES [Set transfer mode]"
            },
            {
              "registers": {
                "command": 239,
                "features": 3,
                "count": 12,
                "lba": 14390272,
                "device": 0,
                "device_control": 0
              },
              "powerup_milliseconds": 3899562,
              "command_name": "SET FEATURES [Set transfer mode]"
            },
            {
              "registers": {
                "command": 236,
                "features": 218,
                "count": 240,
                "lba": 14390272,
                "device": 0,
                "device_control": 0
              },
              "powerup_milliseconds": 3899561,
              "command_name": "IDENTIFY DEVICE"
            },
            {
              "registers": {
                "command": 37,
                "features": 218,
                "count": 240,
                "lba": 752587776,
                "device": 64,
                "device_control": 0
              },
              "powerup_milliseconds": 3868538,
              "command_name": "READ DMA EXT"
            }
          ]
        },
        {
          "error_number": 1,
          "log_index": 0,
          "lifetime_hours": 1984,
          "device_state": {
            "value": 3,
            "string": "active or idle"
          },
          "completion_registers": {
            "error": 64,
            "status": 81,
            "count": 240,
            "lba": 752587896,
            "device": 64,
            "device_control": 0
          },
          "error_description": "Error: UNC 240 sectors at LBA = 0x2cdb9478 = 752587896",
          "previous_commands": [
            {
              "registers": {
                "command": 37,
                "features": 218,
                "count": 240,
                "lba": 752587776,
                "device": 64,
                "device_control": 0
              },
              "powerup_milliseconds": 3868538,
              "command_name": "READ DMA EXT"
            },
            {
              "registers": {
                "command": 37,
                "features": 218,
                "count": 16,
                "lba": 752587760,
                "device": 64,
                "device_control": 0
              },
              "powerup_milliseconds": 3868537,
              "command_name": "READ DMA EXT"
            },
            {
              "registers": {
                "command": 37,
                "features": 218,
                "count": 240,
                "lba": 752587520,
                "device": 64,
                "device_control": 0
              },
              "powerup_milliseconds": 3868534,
              "command_name": "READ DMA EXT"
            },
            {
              "registers": {
                "command": 37,
                "features": 218,
                "count": 16,
                "lba": 752587504,
                "device": 64,
                "device_control": 0
              },
              "powerup_milliseconds": 3868534,
              "command_name": "READ DMA EXT"
            },
            {
              "registers": {
                "command": 37,
                "features": 218,
                "count": 240,
                "lba": 752587264,
                "device": 64,
                "device_control": 0
              },
              "powerup_milliseconds": 3868530,
              "command_name": "READ DMA EXT"
            }
          ]
        }
      ]
    }
  },
  "ata_smart_self_test_log": {
    "extended": {
      "revision": 1,
      "sectors": 1,
      "count": 0
    }
  },
  "ata_smart_selective_self_test_log": {
    "revision": 1,
    "table": [
      {
        "lba_min": 0,
        "lba_max": 0,
        "status": {
          "value": 0,
          "string": "Not_testing"
        }
      },
      {
        "lba_min": 0,
        "lba_max": 0,
        "status": {
          "value": 0,
          "string": "Not_testing"
        }
      },
      {
        "lba_min": 0,
        "lba_max": 0,
        "status": {
          "value": 0,
          "string": "Not_testing"
        }
      },
      {
        "lba_min": 0,
        "lba_max": 0,
        "status": {
          "value": 0,
          "string": "Not_testing"
        }
      },
      {
        "lba_min": 0,
        "lba_max": 0,
        "status": {
          "value": 0,
          "string": "Not_testing"
        }
      }
    ],
    "flags": {
      "value": 0,
      "remainder_scan_enabled": false
    },
    "power_up_scan_resume_minutes": 0
  },
  "ata_sct_status": {
    "format_version": 3,
    "sct_version": 258,
    "device_state": {
      "value": 0,
      "string": "Active"
    },
    "temperature": {
      "current": 41,
      "power_cycle_min": 23,
      "power_cycle_max": 41,
      "lifetime_min": 14,
      "lifetime_max": 50,
      "under_limit_count": 0,
      "over_limit_count": 0
    },
    "vendor_specific": [
      1,
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      0,
      0
    ]
  },
  "ata_sct_temperature_history": {
    "version": 2,
    "sampling_period_minutes": 1,
    "logging_interval_minutes": 1,
    "temperature": {
      "op_limit_min": 0,
      "op_limit_max": 60,
      "limit_min": -41,
      "limit_max": 85
    },
    "size": 128,
    "index": 7,
    "table": [
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41,
      41
    ]
  },
  "sata_phy_event_counters": {
    "table": [
      {
        "id": 1,
        "name": "Command failed due to ICRC error",
        "size": 2,
        "value": 0,
        "overflow": false
      },
      {
        "id": 2,
        "name": "R_ERR response for data FIS",
        "size": 2,
        "value": 0,
        "overflow": false
      },
      {
        "id": 3,
        "name": "R_ERR response for device-to-host data FIS",
        "size": 2,
        "value": 0,
        "overflow": false
      },
      {
        "id": 4,
        "name": "R_ERR response for host-to-device data FIS",
        "size": 2,
        "value": 0,
        "overflow": false
      },
      {
        "id": 5,
        "name": "R_ERR response for non-data FIS",
        "size": 2,
        "value": 0,
        "overflow": false
      },
      {
        "id": 6,
        "name": "R_ERR response for device-to-host non-data FIS",
        "size": 2,
        "value": 0,
        "overflow": false
      },
      {
        "id": 7,
        "name": "R_ERR response for host-to-device non-data FIS",
        "size": 2,
        "value": 0,
        "overflow": false
      },
      {
        "id": 8,
        "name": "Device-to-host non-data FIS retries",
        "size": 2,
        "value": 0,
        "overflow": false
      },
      {
        "id": 9,
        "name": "Transition from drive PhyRdy to drive PhyNRdy",
        "size": 2,
        "value": 0,
        "overflow": false
      },
      {
        "id": 10,
        "name": "Device-to-host register FISes sent due to a COMRESET",
        "size": 2,
        "value": 1,
        "overflow": false
      },
      {
        "id": 11,
        "name": "CRC errors within host-to-device FIS",
        "size": 2,
        "value": 0,
        "overflow": false
      },
      {
        "id": 15,
        "name": "R_ERR response for host-to-device data FIS, CRC",
        "size": 2,
        "value": 0,
        "overflow": false
      },
      {
        "id": 18,
        "name": "R_ERR response for host-to-device non-data FIS, CRC",
        "size": 2,
        "value": 0,
        "overflow": false
      },
      {
        "id": 32768,
        "name": "Vendor specific",
        "size": 4,
        "value": 86780,
        "overflow": false
      }
    ],
    "reset": false
  }
}

smartctl -x --json /dev/sdb:

{
  "json_format_version": [
    1,
    0
  ],
  "smartctl": {
    "version": [
      7,
      1
    ],
    "svn_revision": "5022",
    "platform_info": "armv7l-linux-5.10.60-v7l+",
    "build_info": "(local build)",
    "argv": [
      "smartctl",
      "-x",
      "--json",
      "/dev/sdb"
    ],
    "exit_status": 2
  },
  "device": {
    "name": "/dev/sdb",
    "info_name": "/dev/sdb [SAT]",
    "type": "sat",
    "protocol": "ATA"
  }
}

AnalogJ · 2021-10-27T03:45:53Z

@garret @austozi just to confirm /dev/sdb is a real disk and it was sleeping at the time you each ran smartctl -x --json /dev/sdb correct?

garret · 2021-10-27T06:13:42Z

@AnalogJ Yes, I run the commands after a long time of idle when I haven't accessed anything on the disk for many hours. The way I see that the hdd goes out of sleep is that when trying to access via ssh to its content I get a delay of like 20 seconds to "ls" on it, so the hdd to starts spinning up. I can also hear it.

AnalogJ · 2021-10-28T06:51:43Z

ahh ok. I just took a closer look at that non-zero exit code of 2 and it matches bit 1 in the docs:
http://www.linuxguide.it/command_line/linux-manpage/do.php?file=smartctl#sect7

Return Values

The return values of smartctl are defined by a bitmask. If all is well with the disk, the return value (exit status) of smartctl is 0 (all bits turned off). If a problem occurs, or an error, potential error, or fault is detected, then a non-zero status is returned. In this case, the eight different bits in the return value have the following meanings for ATA disks; some of these values may also be returned for SCSI disks.

Bit 0: Command line did not parse.

Bit 1: Device open failed, or device did not return an IDENTIFY DEVICE structure.

Bit 2: Some SMART command to the disk failed, or there was a checksum error in a SMART data structure (see В´-bВ´ option above).

Bit 3: SMART status check returned “DISK FAILING".

Bit 4: We found prefail Attributes <= threshold.

Bit 5: SMART status check returned “DISK OK” but we found that some (usage or prefail) Attributes have been <= threshold at some time in the past.

Bit 6: The device error log contains records of errors.

Bit 7: The device self-test log contains records of errors.

To test within the shell for whether or not the different bits are turned on or off, you can use the following type of construction (this is bash syntax):
smartstat=$(($? & 8))
This looks at only at bit 3 of the exit status $? (since 8=2^3). The shell variable $smartstat will be nonzero if SMART status check returned “disk failing” and zero otherwise.

In general, scrutiny shouldn't be forwarding empty, invalid data from the collector to the webapp, but we ignore non-zero exit codes because bit 3 will be set when smartctl detects an error, and that response is valid, with real meaningful data.

I'll need to come up with a generic process to handle disks in standby mode or that haven't been "seen" in a while -- maybe just a new notification rule. Atleast I know what to look for now. Thanks for all the data everyone! 👍

michaelkrieger · 2022-01-12T22:06:01Z

Just to add to the logs and provide some suggestions. This is a sleeping external USB hard drive. The hard drive appeared to be too slow to respond to smartctl in its spin up

Suggestion 1: Include a quick read from the drive before running smartctl
# dd if=/dev/sda bs=1k count=1 of=/dev/zero
This will cause the drive to wake up ahead of running smartctl. This should be safe on all drives and will block scrutiny pending the read being successful.

Suggestion 2: Include a probe of the drives before running smartctl
# partprobe -d /dev/sda
This should cause the drive to wake up.

Suggestion 3: Work around
It would appear that if smartctl returns that particular error, the app should sleep for 10 seconds and the command should simply be retried.

Recommendation
I'd probably do one of the first two, as there is little downside in a quick read/probe prior to running smartctl. It would ensure it always has a live drive to read from.

Additional Enhancement
Irrespective, it should also maybe not add the drive to the dashboard if the drive wasn't detected properly.

Logs showing one sleeping drive (sdc)

**# scrutiny-collector-metrics run**
2022/01/12 16:41:36 Loading configuration file: /scrutiny/config/collector.yaml

 ___   ___  ____  __  __  ____  ____  _  _  _  _
/ __) / __)(  _ \(  )(  )(_  _)(_  _)( \( )( \/ )
\__ \( (__  )   / )(__)(   )(   _)(_  )  (  \  /
(___/ \___)(_)\_)(______) (__) (____)(_)\_) (__)
AnalogJ/scrutiny/metrics                               dev-0.3.12

INFO[0000] Verifying required tools                      type=metrics
INFO[0000] Executing command: smartctl --scan -j         type=metrics
INFO[0000] Executing command: smartctl --info -j -d sat /dev/sdb  type=metrics
INFO[0000] Generating WWN                                type=metrics
INFO[0000] Executing command: smartctl --info -j -d sat /dev/sdc  type=metrics
**ERRO[0008] Could not retrieve device information for sdc: exit status 2  type=metrics**
INFO[0008] Executing command: smartctl --info -j -d sat /dev/sda  type=metrics
INFO[0009] Generating WWN                                type=metrics
INFO[0009] Sending detected devices to API, for filtering & validation  type=metrics
INFO[0009] Collecting smartctl results for sdb           type=metrics
INFO[0009] Executing command: smartctl -x -j -d sat /dev/sdb  type=metrics
INFO[0009] Publishing smartctl results for ...  type=metrics
INFO[0009] Collecting smartctl results for sdc           type=metrics
INFO[0009] Executing command: smartctl -x -j -d sat /dev/sdc  type=metrics
**ERRO[0013] smartctl returned an error code (2) while processing sdc  type=metrics**
**ERRO[0013] smartctl could not open device                type=metrics**
INFO[0013] Publishing smartctl results for               type=metrics
INFO[0013] Collecting smartctl results for sda           type=metrics
INFO[0013] Executing command: smartctl -x -j -d sat /dev/sda  type=metrics
INFO[0013] Publishing smartctl results for ...  type=metrics
INFO[0013] Main: Completed                               type=metrics

**# scrutiny-collector-metrics run**
2022/01/12 16:41:58 Loading configuration file: /scrutiny/config/collector.yaml

 ___   ___  ____  __  __  ____  ____  _  _  _  _
/ __) / __)(  _ \(  )(  )(_  _)(_  _)( \( )( \/ )
\__ \( (__  )   / )(__)(   )(   _)(_  )  (  \  /
(___/ \___)(_)\_)(______) (__) (____)(_)\_) (__)
AnalogJ/scrutiny/metrics                               dev-0.3.12

INFO[0000] Verifying required tools                      type=metrics
INFO[0000] Executing command: smartctl --scan -j         type=metrics
INFO[0000] Executing command: smartctl --info -j -d sat /dev/sda  type=metrics
INFO[0000] Generating WWN                                type=metrics
INFO[0000] Executing command: smartctl --info -j -d sat /dev/sdb  type=metrics
INFO[0000] Generating WWN                                type=metrics
INFO[0000] Executing command: smartctl --info -j -d sat /dev/sdc  type=metrics
INFO[0000] Generating WWN                                type=metrics
INFO[0000] Sending detected devices to API, for filtering & validation  type=metrics
INFO[0000] Collecting smartctl results for sda           type=metrics
INFO[0000] Executing command: smartctl -x -j -d sat /dev/sda  type=metrics
INFO[0000] Publishing smartctl results for ... type=metrics
INFO[0000] Collecting smartctl results for sdb           type=metrics
INFO[0000] Executing command: smartctl -x -j -d sat /dev/sdb  type=metrics
INFO[0001] Publishing smartctl results for ... type=metrics
INFO[0001] Collecting smartctl results for sdc           type=metrics
INFO[0001] Executing command: smartctl -x -j -d sat /dev/sdc  type=metrics
INFO[0001] Publishing smartctl results for ... type=metrics
INFO[0001] Main: Completed                               type=metrics
\```

</details>

edfield · 2023-01-27T03:39:55Z

Having the same problem on an Unraid server. When the SAS drives are spun down and the smart read is triggered, it throws read errors in the logs. If it goes on long enough the phantom drives start appearing leading to disks being deactivated. No damage done as long as I catch it before a 3rd disk is deactivated, just an 18hr data rebuild. If disk spindown is disabled everything works fine. Since unraid logs are dynamic, I don't currently have any to upload. I'll see if I can get it to do it again and capture a diagnostic image after the current rebuild finishes if you need them.

AnalogJ · 2023-01-27T03:55:06Z

Hey @edfield are you saying that running Smartctl/Scrutiny is causing corruption/errors in your RAID causing a re-build?
If so, thats a huge problem and I'll need to add a warning in the README.

edfield · 2023-01-27T04:03:35Z

Not corruption, but after a number of failed reads, unraid will deactivate the drive. To get the drive back in the array takes a data rebuild. I run 2 parity drives so 2 drives can be deactivated with no loss, a third would be real bad. I have 14 data drives so quite a few are spun down most of the time. All drives are SAS enterprise drives with 2 adaptec SAS controllers.

garret added the bug Something isn't working label Apr 9, 2021

AnalogJ closed this as completed Apr 24, 2021

AnalogJ reopened this Apr 24, 2021

AnalogJ added the waiting for response label Apr 24, 2021

AnalogJ added enhancement New feature or request and removed waiting for response labels Oct 28, 2021

AnalogJ changed the title ~~[BUG] Scrutiny sees my two disks but after some time also shows another empty disk~~ [BUG] Disks in standby mode are not correctly processed/detected - Scrutiny will create a duplicate disk with empty data Oct 28, 2021

AnalogJ changed the title ~~[BUG] Disks in standby mode are not correctly processed/detected - Scrutiny will create a duplicate disk with empty data~~ [Standby-Support][BUG] Disks in standby mode are not correctly processed/detected - Scrutiny will create a duplicate disk with empty data Jun 1, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Standby-Support][BUG] Disks in standby mode are not correctly processed/detected - Scrutiny will create a duplicate disk with empty data #157

[Standby-Support][BUG] Disks in standby mode are not correctly processed/detected - Scrutiny will create a duplicate disk with empty data #157

garret commented Apr 9, 2021

AnalogJ commented Apr 24, 2021

AnalogJ commented Apr 24, 2021

AnalogJ commented Apr 24, 2021

garret commented Apr 25, 2021

ngkengwooi commented Aug 16, 2021

garret commented Aug 17, 2021

ngkengwooi commented Sep 12, 2021 •

edited

mark-monteiro commented Sep 12, 2021

AnalogJ commented Oct 23, 2021

garret commented Oct 25, 2021

AnalogJ commented Oct 26, 2021

garret commented Oct 26, 2021

austozi commented Oct 26, 2021

AnalogJ commented Oct 27, 2021 •

edited

garret commented Oct 27, 2021

AnalogJ commented Oct 28, 2021

michaelkrieger commented Jan 12, 2022 •

edited

edfield commented Jan 27, 2023 •

edited

AnalogJ commented Jan 27, 2023

edfield commented Jan 27, 2023 •

edited

[Standby-Support][BUG] Disks in standby mode are not correctly processed/detected - Scrutiny will create a duplicate disk with empty data #157

[Standby-Support][BUG] Disks in standby mode are not correctly processed/detected - Scrutiny will create a duplicate disk with empty data #157

Comments

garret commented Apr 9, 2021

AnalogJ commented Apr 24, 2021

AnalogJ commented Apr 24, 2021

AnalogJ commented Apr 24, 2021

garret commented Apr 25, 2021

ngkengwooi commented Aug 16, 2021

garret commented Aug 17, 2021

ngkengwooi commented Sep 12, 2021 • edited

mark-monteiro commented Sep 12, 2021

AnalogJ commented Oct 23, 2021

garret commented Oct 25, 2021

AnalogJ commented Oct 26, 2021

garret commented Oct 26, 2021

austozi commented Oct 26, 2021

AnalogJ commented Oct 27, 2021 • edited

garret commented Oct 27, 2021

AnalogJ commented Oct 28, 2021

michaelkrieger commented Jan 12, 2022 • edited

edfield commented Jan 27, 2023 • edited

AnalogJ commented Jan 27, 2023

edfield commented Jan 27, 2023 • edited

ngkengwooi commented Sep 12, 2021 •

edited

AnalogJ commented Oct 27, 2021 •

edited

michaelkrieger commented Jan 12, 2022 •

edited

edfield commented Jan 27, 2023 •

edited

edfield commented Jan 27, 2023 •

edited