Storage managment API: disks infos by christophehenry · Pull Request #1953 · YunoHost/yunohost

christophehenry · 2024-09-17T15:27:59Z

The problem

Rel: YunoHost/issues#1823.
Add API for getting infos on hard drives.

christophehenry · 2024-09-18T15:55:21Z

Ok I think this is a good first step. Here is an exemple of the result this produces:

{
    "sda": {
        "devname": "/dev/sda",
        "model": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
        "serial": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
        "size": xxxxxxxxxxxxxxxxxxxxxxxxxxxxx,
        "links": [
            "/dev/disk/by-diskseq/1",
            "/dev/disk/by-id/ata-xxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
            "/dev/disk/by-id/wwn-xxxxxxxxxxxxxxxxxxxxxxxx",
            "/dev/disk/by-path/pci-0000:01:00.1-ata-1",
            "/dev/disk/by-path/pci-0000:01:00.1-ata-1.0"
        ],
        "partitions": {
            "sda1": {
                "devname": "/dev/sda",
                "filesystem": "btrfs",
                "encrypted": true,
                "mountpoint": "/home"
            }
        }
    },
    "sdb": {
        "devname": "/dev/sdb",
        "model": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
        "serial": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
        "size": xxxxxxxxxxxxxxxxxxxxxxxxxxxxx,
        "links": [
            "/dev/disk/by-diskseq/2",
            "/dev/disk/by-id/ata-xxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
            "/dev/disk/by-id/wwn-xxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
            "/dev/disk/by-path/pci-0000:0a:00.0-ata-6",
            "/dev/disk/by-path/pci-0000:0a:00.0-ata-6.0"
        ],
        "partitions": {
            "sdb1": {
                "devname": "/dev/sdb",
                "filesystem": "vfat",
                "encrypted": false,
                "mountpoint": "/boot/efi"
            },
            "sdb2": {
                "devname": "/dev/sdb",
                "filesystem": "ext4",
                "encrypted": false,
                "mountpoint": "/boot"
            },
            "sdb3": {
                "devname": "/dev/sdb",
                "filesystem": "btrfs",
                "encrypted": true,
                "mountpoint": "/"
            }
        }
    }
}

In a future PR, I'd like to add the device type (HDD, SSD, …) and RPM for HDDs. I don't really know how to do that. TrueNAS does it by using its own library. In the meantime, this is ready for review.

christophehenry · 2024-09-29T14:36:32Z

A few notes here: I rewrote this PR to use the udisks2 API which provides much more details about disks and is easier to use that udev + other sources of infos.

But as a pure dbus API, the code is less self-explanatory. It's mainly about parsing dictionary.

Maybe I can make use of something like dbus-fast, IDK.

Tell me if it's OK to add the udisk daemon to Ynh. I'm not aware of everything it entails.

alexAubin · 2024-09-29T14:43:46Z

Naively udisk sounds fine to me 👍

zamentur · 2024-10-13T14:54:20Z

Yes i think udisk is a good idea, also because it could help to automount external usb or disk...

zamentur · 2024-10-13T15:12:42Z

In a future PR, I'd like to add the device type (HDD, SSD, …) and RPM for HDDs. I don't really know how to do that. TrueNAS does it by using its own library. In the meantime, this is ready for review.

You can know if it's a HDD or SSD by checking directly the info inside /sys...

cat /sys/block/sda/queue/rotational

0 -> SSD
1 -> HDD

/sys/block/sda/queue/removable to know if it's an usb removable disk i guess

We could also imagine check for ssd how many written is made, in order to compute an end of life percentage.

christophehenry · 2024-10-14T09:09:34Z

You can know if it's a HDD or SSD by checking directly the info inside /sys...

udisks2 provides the information so I don't need a solution anymore. Thanks.

christophehenry · 2025-01-19T10:54:18Z

+    for k, v in manager.get_dbus_method(
+        "GetManagedObjects", "org.freedesktop.DBus.ObjectManager"
+    )().items():


Does someone knows a more efficient way to interact with dbus than just list all the objects managed by Udisks2 here?

christophehenry · 2025-01-19T11:04:07Z

I don't believe it is useful to test this. We'd basically be testing the udisks2 library which is never a good idea.

zamentur

I suggest to split into 2 commands

yunohos storage disk list
yunohos storage disk info <disk>

zamentur · 2025-01-28T09:41:57Z

 , python3-ldap, python3-zeroconf (>= 0.47), python3-lexicon,
 , python3-cryptography, python3-jwt, python3-passlib, python3-magic
 , python-is-python3, python3-pydantic, python3-email-validator
+ , udisks2,


We might add udisks2 suggested package to be sure those package will be installed on arm.

Suggested change

, udisks2,

, udisks2, udisks2-bcache udisks2-btrfs udisks2-lvm2 udisks2-zram

zamentur · 2025-01-28T10:45:59Z

+      subcategory_help: Manage et get infos about hard-drives
+      actions:
+        # storage_disks_list
+        infos:


I suggest to be more consistent with the existing other command lines by splitting in 2 commands:

$ yunohost storage disk list [--with-info [--human-readable]] $ yunohost storage disk info <disk_name> [--human-readable] name: XXXX model: YYYY serial: ZZZZ size: 120GB type: SSD rpm: None

About --human-readable we are doing this in several part of the app, but regarding the cli is often use by a human, i ask me if we should not put by default the --human-readable and configure an other option to get the size in Bytes...

zamentur · 2025-01-28T10:49:14Z

+            "size": drive["Size"],
+            "type": "HDD" if rotation_rate != 0 else "SSD",
+            "rpm": rotation_rate if rotation_rate != 0 else None,
+        }


What if there are no disks at all ? We probably should display a message or something. Currently, it's just like this:

root@ynh:/ynh-dev# yunohost storage disk infos root@ynh:/ynh-dev#

When using the --human-readable option you mean?

With --human-readable :

size: 60 GB

Without:

size: 64424509440

No I meant about the case where there's no disks at all. With --human-readable, a message, without, empty list. Is that OK?

zamentur · 2025-01-28T10:55:16Z

+            "model": drive["Model"],
+            "serial": drive["Serial"],
+            "size": drive["Size"],
+            "type": "HDD" if rotation_rate != 0 else "SSD",


I have tested on my computer with an USB key, and i got -1

Suggested change

"type": "HDD" if rotation_rate != 0 else "SSD",

"type": "HDD" if rotation_rate <= 0 else "SSD",

I made this suggestion, but we probably should distinguish USB key, SSD, HDD, SSHD (Nvme ?)

zamentur · 2025-01-28T10:59:01Z

+            "name": name,
+            "model": drive["Model"],
+            "serial": drive["Serial"],
+            "size": drive["Size"],


We probably need the accurate value in B and a more human value.

zamentur · 2025-01-28T11:04:08Z

I tried to test into an incus container, but drives are not discovered by dbus (lsblk does). It's due to the fact that the main storage is on /dev/loopXX on incus. I tried to add an entire usb key, but it was a fail.

christophehenry · 2025-02-01T10:41:15Z

@zamentur So I implemented you suggestions and tested with a USB flash drive, USB hard drive and SD card.

Here are the results

With --with-info

{
    # This is the SD card
    "******************": {
        "name": "******************",
        "model": "*****************",
        "serial": "*****************",
        "removable": true,
        "size": "3.6 GB",
        "connection_bus": "sdio",
        "type": "SSD"
    },
    # This is an internal SSD
    "******************": {
        "name": "******************",
        "model": "*****************",
        "serial": "*****************",
        "removable": false,
        "size": "223.6 GB",
        "type": "SSD"
    },
    # This is the USB HDD
    "******************": {
        "name": "******************",
        "model": "*****************",
        "serial": "*****************",
        "removable": false,
        "size": "1.8 TB",
        "connection_bus": "usb",
        "type": "HDD",
        "rpm": 5400
    },
    # This is an internal SSD
    "******************": {
        "name": "******************",
        "model": "*****************",
        "serial": "*****************",
        "removable": false,
        "size": "1.8 TB",
        "type": "SSD"
    },
    # This is an USB flash drive
    "******************": {
        "name": "******************",
        "model": "*****************",
        "serial": "*****************",
        "removable": true,
        "size": "7.2 GB",
        "connection_bus": "usb",
        "type": "HDD",
        "rpm": "Unknown"
    }
}

Without --with-info just returns the keys of the dict above.

Weirdly, the USB flash drive is detected as being rotational with unkown RPM. I don't know how this should be handled.

zamentur · 2025-02-01T21:57:16Z

+    if human_readable and not result:
+        return "No external media found"
+
+    return result


Suggested change

return result

return {"disks": result}

It's done like that in user_list backup_list and permission_list, with disks: no need to add a message to explain that there is no disks at all i guess.

Ok, then I guess result should be a list and not a dict. What do you say?

christophehenry · 2025-02-16T15:48:39Z

    ACTIONS = [
        action
-        for action in OPTION_SUBTREE[category]["actions"].keys()
+        for action in OPTION_SUBTREE[category].get("actions", {}).keys()


Il n'y a pas de section action dans storage pour le moment.

zamentur · 2025-02-18T18:36:57Z

We probably should merge into a temporary branch to run all CI tests...

christophehenry commented Sep 17, 2024

View reviewed changes

Comment thread src/tests/test_storage.py Outdated

christophehenry force-pushed the storage-api-disks-infos branch from cb97422 to 6dbe2e1 Compare September 17, 2024 15:29

christophehenry mentioned this pull request Sep 17, 2024

Add storage management section + disks management page YunoHost/yunohost-admin#585

Merged

christophehenry commented Sep 17, 2024

View reviewed changes

Comment thread share/actionsmap.yml Outdated

christophehenry force-pushed the storage-api-disks-infos branch from 6dbe2e1 to a2b7e81 Compare September 18, 2024 15:48

christophehenry marked this pull request as ready for review September 18, 2024 15:50

christophehenry force-pushed the storage-api-disks-infos branch from a2b7e81 to a1a394f Compare September 18, 2024 15:50

christophehenry commented Sep 18, 2024

View reviewed changes

Comment thread src/disks.py Outdated

christophehenry changed the title ~~WIP: Storage mangment API: disks infos~~ Storage mangment API: disks infos Sep 18, 2024

christophehenry force-pushed the storage-api-disks-infos branch 2 times, most recently from 81d9fa7 to e83e7b3 Compare September 22, 2024 15:05

alexAubin added the 🏗️ Major project Big decision label Oct 15, 2024

christophehenry force-pushed the storage-api-disks-infos branch 8 times, most recently from 7d9c4de to 06e2219 Compare January 18, 2025 14:51

christophehenry commented Jan 19, 2025

View reviewed changes

Comment thread src/disks.py Outdated

christophehenry force-pushed the storage-api-disks-infos branch from 06e2219 to bcfd1ae Compare January 19, 2025 11:35

christophehenry force-pushed the storage-api-disks-infos branch from bcfd1ae to ea7051a Compare January 19, 2025 11:53

christophehenry commented Jan 22, 2025

View reviewed changes

Comment thread src/disks.py Outdated

christophehenry force-pushed the storage-api-disks-infos branch from ea7051a to 0961615 Compare January 22, 2025 13:14

zamentur requested changes Jan 28, 2025

View reviewed changes

christophehenry force-pushed the storage-api-disks-infos branch 2 times, most recently from d3c3f61 to 0877f01 Compare February 1, 2025 10:31

christophehenry commented Feb 1, 2025

View reviewed changes

Comment thread src/disks.py

christophehenry force-pushed the storage-api-disks-infos branch from 0877f01 to 45e30e6 Compare February 1, 2025 10:50

Storage mangment API: disks infos

7f659f7

christophehenry force-pushed the storage-api-disks-infos branch from 45e30e6 to 7f659f7 Compare February 1, 2025 10:51

zamentur reviewed Feb 1, 2025

View reviewed changes

Comment thread src/disks.py Outdated

zamentur reviewed Feb 1, 2025

View reviewed changes

zamentur changed the title ~~Storage mangment API: disks infos~~ Storage managment API: disks infos Feb 2, 2025

christophehenry commented Feb 2, 2025

View reviewed changes

Comment thread src/disks.py

christophehenry force-pushed the storage-api-disks-infos branch from 9a14fcf to 00b390a Compare February 2, 2025 15:33

christophehenry force-pushed the storage-api-disks-infos branch 2 times, most recently from d2ea214 to 34b801f Compare February 16, 2025 15:48

christophehenry commented Feb 16, 2025

View reviewed changes

christophehenry force-pushed the storage-api-disks-infos branch 2 times, most recently from 83b05ba to f22365d Compare February 16, 2025 19:24

github-advanced-security AI found potential problems Feb 18, 2025

View reviewed changes

Comment thread src/disks.py Fixed

Apply suggestions from peer review

d2a40cd

christophehenry force-pushed the storage-api-disks-infos branch from f22365d to d2a40cd Compare February 18, 2025 16:16

zamentur approved these changes Feb 18, 2025

View reviewed changes

zamentur merged commit b5afc8c into YunoHost:dev Feb 18, 2025

christophehenry deleted the storage-api-disks-infos branch February 19, 2025 14:17

	, udisks2,
	, udisks2, udisks2-bcache udisks2-btrfs udisks2-lvm2 udisks2-zram

	"type": "HDD" if rotation_rate != 0 else "SSD",
	"type": "HDD" if rotation_rate <= 0 else "SSD",

Uh oh!

Conversation

christophehenry commented Sep 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The problem

Uh oh!

Uh oh!

Uh oh!

christophehenry commented Sep 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

christophehenry commented Sep 29, 2024

Uh oh!

alexAubin commented Sep 29, 2024

Uh oh!

zamentur commented Oct 13, 2024

Uh oh!

zamentur commented Oct 13, 2024

Uh oh!

christophehenry commented Oct 14, 2024

Uh oh!

christophehenry Jan 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

christophehenry commented Jan 19, 2025

Uh oh!

Uh oh!

zamentur left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zamentur commented Jan 28, 2025

Uh oh!

christophehenry commented Feb 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zamentur Feb 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

christophehenry Feb 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

zamentur commented Feb 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

christophehenry commented Sep 17, 2024 •

edited

Loading

christophehenry commented Sep 18, 2024 •

edited

Loading

christophehenry Jan 19, 2025 •

edited

Loading

christophehenry commented Feb 1, 2025 •

edited

Loading

zamentur Feb 1, 2025 •

edited

Loading

christophehenry Feb 2, 2025 •

edited

Loading