Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

md raid as metadataDevice not working #6715

Closed
chrono2002 opened this issue Nov 28, 2020 · 6 comments
Closed

md raid as metadataDevice not working #6715

chrono2002 opened this issue Nov 28, 2020 · 6 comments
Labels
bug wontfix

Comments

@chrono2002
Copy link

chrono2002 commented Nov 28, 2020 •
edited

Hello
I have several machines in cluster with SSD+SSD in md raid and ATA disks.
I'm trying to setup rook-ceph with metadataDevice on SSD md raid and OSDs on ATA disks like this:

  storage:
    useAllDevices: false
    useAllNodes: true
    devicePathFilter: "^/dev/disk/by-id/ata-ST.*"
    config:
      metadataDevice: "md0p3"

Looks like it's skipping md devices somehow. However it's not reliable to keep metadata on raw device cause single drive failure could destroy all data. Is it possible to use md raid as metadata device? Or there is another reliable solution?

sd-prepare log:

2020-11-28 14:31:15.680911 I | rookcmd: starting Rook v1.5.1 with arguments '/rook/rook ceph osd provision'
2020-11-28 14:31:15.681141 I | rookcmd: flag values: --cluster-id=9850d371-0108-45b1-822b-07536465547b, --data-device-filter=, --data-device-path-filter=^/dev/disk/by-id/ata-ST.*, --data-devices=, --drive-groups=, --encrypted-device=false, --force-format=false, --help=false, --location=, --log-flush-frequency=5s, --log-level=DEBUG, --metadata-device=md0p3, --node-name=hosting617717, --operator-image=, --osd-database-size=0, --osd-store=, --osd-wal-size=576, --osds-per-device=1, --pvc-backed-osd=false, --service-account=
2020-11-28 14:31:15.681254 I | op-mon: parsing mon endpoints: c=10.220.223.205:6789,a=10.220.93.72:6789,b=10.220.97.126:6789
2020-11-28 14:31:15.694212 I | op-osd: CRUSH location=root=default host=hosting617717
2020-11-28 14:31:15.694233 I | cephcmd: crush location of osd: root=default host=hosting617717
2020-11-28 14:31:15.694251 D | exec: Running command: nsenter --mount=/rootfs/proc/1/ns/mnt -- /usr/sbin/lvm --help
2020-11-28 14:31:15.705134 I | cephosd: successfully called nsenter
2020-11-28 14:31:15.705157 I | cephosd: binary "/usr/sbin/lvm" found on the host, proceeding with osd preparation
2020-11-28 14:31:15.705164 D | exec: Running command: dmsetup version
2020-11-28 14:31:15.707318 I | cephosd: Library version:   1.02.169-RHEL8 (2020-02-11)
Driver version:    4.42.0
2020-11-28 14:31:15.716651 D | cephclient: No ceph configuration override to merge as "rook-config-override" configmap is empty
2020-11-28 14:31:15.716679 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2020-11-28 14:31:15.716811 I | cephclient: generated admin config in /var/lib/rook/rook-ceph
2020-11-28 14:31:15.716972 D | cephosd: config file @ /etc/ceph/ceph.conf: [global]
fsid                = a53c7a93-60cb-42c7-ad45-2b7cfd9fb1a0
mon initial members = c a b
mon host            = [v2:10.220.223.205:3300,v1:10.220.223.205:6789],[v2:10.220.93.72:3300,v1:10.220.93.72:6789],[v2:10.220.97.126:3300,v1:10.220.97.126:6789]
public addr         = 10.30.210.178
cluster addr        = 10.30.210.178

[client.admin]
keyring = /var/lib/rook/rook-ceph/client.admin.keyring

2020-11-28 14:31:15.716982 I | cephosd: discovering hardware
2020-11-28 14:31:15.716989 D | exec: Running command: lsblk --all --noheadings --list --output KNAME
2020-11-28 14:31:15.720704 D | exec: Running command: lsblk /dev/sda --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-28 14:31:15.722588 D | exec: Running command: sgdisk --print /dev/sda
2020-11-28 14:31:15.728945 D | exec: Running command: udevadm info --query=property /dev/sda
2020-11-28 14:31:15.734907 D | exec: Running command: lsblk --noheadings --pairs /dev/sda
2020-11-28 14:31:15.738843 D | exec: Running command: lsblk /dev/sdb --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-28 14:31:15.740650 D | exec: Running command: sgdisk --print /dev/sdb
2020-11-28 14:31:15.744433 D | exec: Running command: udevadm info --query=property /dev/sdb
2020-11-28 14:31:15.749730 D | exec: Running command: lsblk --noheadings --pairs /dev/sdb
2020-11-28 14:31:15.753716 D | exec: Running command: lsblk /dev/sdc --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-28 14:31:15.755832 D | exec: Running command: sgdisk --print /dev/sdc
2020-11-28 14:31:15.758183 D | exec: Running command: udevadm info --query=property /dev/sdc
2020-11-28 14:31:15.763772 D | exec: Running command: lsblk --noheadings --pairs /dev/sdc
2020-11-28 14:31:15.768409 I | inventory: skipping device "sdc" because it has child, considering the child instead.
2020-11-28 14:31:15.768453 D | exec: Running command: lsblk /dev/sdc1 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-28 14:31:15.770352 D | exec: Running command: udevadm info --query=property /dev/sdc1
2020-11-28 14:31:15.775801 D | exec: Running command: lsblk /dev/sdd --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-28 14:31:15.777738 D | exec: Running command: sgdisk --print /dev/sdd
2020-11-28 14:31:15.780234 W | inventory: skipping device "sdd". exit status 2
2020-11-28 14:31:15.780253 D | exec: Running command: lsblk /dev/sdd1 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-28 14:31:15.782179 D | exec: Running command: udevadm info --query=property /dev/sdd1
2020-11-28 14:31:15.787443 D | exec: Running command: lsblk /dev/md0 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-28 14:31:15.790455 W | inventory: skipping device "md0". unsupported diskType raid1
2020-11-28 14:31:15.790470 D | exec: Running command: lsblk /dev/md0 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-28 14:31:15.792607 W | inventory: skipping device "md0". unsupported diskType raid1
2020-11-28 14:31:15.792660 D | exec: Running command: lsblk /dev/nbd0 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-28 14:31:15.794484 W | inventory: skipping device "nbd0". diskType is empty
2020-11-28 14:31:15.794503 D | exec: Running command: lsblk /dev/nbd1 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-28 14:31:15.796414 W | inventory: skipping device "nbd1". diskType is empty
2020-11-28 14:31:15.796433 D | exec: Running command: lsblk /dev/nbd2 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-28 14:31:15.798135 W | inventory: skipping device "nbd2". diskType is empty
2020-11-28 14:31:15.798155 D | exec: Running command: lsblk /dev/nbd3 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-28 14:31:15.799932 W | inventory: skipping device "nbd3". diskType is empty
2020-11-28 14:31:15.799951 D | exec: Running command: lsblk /dev/nbd4 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-28 14:31:15.802125 W | inventory: skipping device "nbd4". diskType is empty
2020-11-28 14:31:15.802145 D | exec: Running command: lsblk /dev/nbd5 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-28 14:31:15.803853 W | inventory: skipping device "nbd5". diskType is empty
2020-11-28 14:31:15.803867 D | exec: Running command: lsblk /dev/nbd6 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-28 14:31:15.805608 W | inventory: skipping device "nbd6". diskType is empty
2020-11-28 14:31:15.805629 D | exec: Running command: lsblk /dev/nbd7 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-28 14:31:15.807348 W | inventory: skipping device "nbd7". diskType is empty
2020-11-28 14:31:15.807368 D | exec: Running command: lsblk /dev/md0p1 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-28 14:31:15.809269 W | inventory: skipping device "md0p1". unsupported diskType md
2020-11-28 14:31:15.809289 D | exec: Running command: lsblk /dev/md0p1 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-28 14:31:15.811178 W | inventory: skipping device "md0p1". unsupported diskType md
2020-11-28 14:31:15.811206 D | exec: Running command: lsblk /dev/md0p2 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-28 14:31:15.813094 W | inventory: skipping device "md0p2". unsupported diskType md
2020-11-28 14:31:15.813112 D | exec: Running command: lsblk /dev/md0p2 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-28 14:31:15.814974 W | inventory: skipping device "md0p2". unsupported diskType md
2020-11-28 14:31:15.814991 D | exec: Running command: lsblk /dev/md0p3 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-28 14:31:15.816910 W | inventory: skipping device "md0p3". unsupported diskType md
2020-11-28 14:31:15.816930 D | exec: Running command: lsblk /dev/md0p3 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-28 14:31:15.819256 W | inventory: skipping device "md0p3". unsupported diskType md
2020-11-28 14:31:15.819274 D | exec: Running command: lsblk /dev/nbd8 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-28 14:31:15.821192 W | inventory: skipping device "nbd8". diskType is empty
2020-11-28 14:31:15.821212 D | exec: Running command: lsblk /dev/nbd9 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-28 14:31:15.823093 W | inventory: skipping device "nbd9". diskType is empty
2020-11-28 14:31:15.823112 D | exec: Running command: lsblk /dev/nbd10 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-28 14:31:15.824941 W | inventory: skipping device "nbd10". diskType is empty
2020-11-28 14:31:15.824971 D | exec: Running command: lsblk /dev/nbd11 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-28 14:31:15.827010 W | inventory: skipping device "nbd11". diskType is empty
2020-11-28 14:31:15.827065 D | exec: Running command: lsblk /dev/nbd12 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-28 14:31:15.828948 W | inventory: skipping device "nbd12". diskType is empty
2020-11-28 14:31:15.828967 D | exec: Running command: lsblk /dev/nbd13 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-28 14:31:15.830966 W | inventory: skipping device "nbd13". diskType is empty
2020-11-28 14:31:15.830985 D | exec: Running command: lsblk /dev/nbd14 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-28 14:31:15.833007 W | inventory: skipping device "nbd14". diskType is empty
2020-11-28 14:31:15.833027 D | exec: Running command: lsblk /dev/nbd15 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-28 14:31:15.834831 W | inventory: skipping device "nbd15". diskType is empty
2020-11-28 14:31:15.834862 D | inventory: discovered disks are [0xc0003ef320 0xc0003ef7a0 0xc00019dd40 0xc00017a480]
2020-11-28 14:31:15.834868 I | cephosd: creating and starting the osds
2020-11-28 14:31:15.841165 D | cephosd: No Drive Groups configured.
2020-11-28 14:31:15.841199 D | cephosd: desiredDevices are [{Name:^/dev/disk/by-id/ata-ST.* OSDsPerDevice:1 MetadataDevice: DatabaseSizeMB:0 DeviceClass: IsFilter:false IsDevicePathFilter:true}]
2020-11-28 14:31:15.841207 D | cephosd: context.Devices are [0xc0003ef320 0xc0003ef7a0 0xc00019dd40 0xc00017a480]
2020-11-28 14:31:15.841215 D | exec: Running command: lsblk /dev/sda --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-28 14:31:15.843098 D | exec: Running command: ceph-volume inventory --format json /dev/sda
2020-11-28 14:31:16.502800 I | cephosd: device "sda" is available.
2020-11-28 14:31:16.502915 I | cephosd: device "sda" (aliases: "/dev/disk/by-id/wwn-0x5000c500932fac52 /dev/disk/by-id/ata-ST8000NM0055-1RM112_ZA15A14X /dev/disk/by-path/pci-0000:05:00.0-sas-phy0-lun-0") matches device path filter "^/dev/disk/by-id/ata-ST.*"
2020-11-28 14:31:16.502927 I | cephosd: device "sda" is selected by the device filter/name "^/dev/disk/by-id/ata-ST.*"
2020-11-28 14:31:16.502947 D | exec: Running command: lsblk /dev/sdb --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2020-11-28 14:31:16.505561 D | exec: Running command: ceph-volume inventory --format json /dev/sdb
2020-11-28 14:31:17.130993 I | cephosd: device "sdb" is available.
2020-11-28 14:31:17.131063 I | cephosd: device "sdb" (aliases: "/dev/disk/by-id/ata-ST8000NM0055-1RM112_ZA15M9Y7 /dev/disk/by-id/wwn-0x5000c50093484442 /dev/disk/by-path/pci-0000:05:00.0-sas-phy1-lun-0") matches device path filter "^/dev/disk/by-id/ata-ST.*"
2020-11-28 14:31:17.131071 I | cephosd: device "sdb" is selected by the device filter/name "^/dev/disk/by-id/ata-ST.*"
2020-11-28 14:31:17.131080 I | cephosd: skipping device "sdc1" because it contains a filesystem "linux_raid_member"
2020-11-28 14:31:17.131085 I | cephosd: skipping device "sdd1" because it contains a filesystem "linux_raid_member"
2020-11-28 14:31:17.131226 I | cephosd: configuring osd devices: {"Entries":{"sda":{"Data":-1,"Metadata":null,"Config":{"Name":"^/dev/disk/by-id/ata-ST.*","OSDsPerDevice":1,"MetadataDevice":"","DatabaseSizeMB":0,"DeviceClass":"","IsFilter":false,"IsDevicePathFilter":true},"PersistentDevicePaths":["/dev/disk/by-id/wwn-0x5000c500932fac52","/dev/disk/by-id/ata-ST8000NM0055-1RM112_ZA15A14X","/dev/disk/by-path/pci-0000:05:00.0-sas-phy0-lun-0"]},"sdb":{"Data":-1,"Metadata":null,"Config":{"Name":"^/dev/disk/by-id/ata-ST.*","OSDsPerDevice":1,"MetadataDevice":"","DatabaseSizeMB":0,"DeviceClass":"","IsFilter":false,"IsDevicePathFilter":true},"PersistentDevicePaths":["/dev/disk/by-id/ata-ST8000NM0055-1RM112_ZA15M9Y7","/dev/disk/by-id/wwn-0x5000c50093484442","/dev/disk/by-path/pci-0000:05:00.0-sas-phy1-lun-0"]}}}
2020-11-28 14:31:17.131299 I | cephclient: getting or creating ceph auth key "client.bootstrap-osd"
2020-11-28 14:31:17.131491 D | exec: Running command: ceph auth get-or-create-key client.bootstrap-osd mon allow profile bootstrap-osd --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --name=client.admin --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json --out-file /tmp/472627608
2020-11-28 14:31:17.695956 I | cephosd: configuring new device sda
2020-11-28 14:31:17.695983 I | cephosd: using md0p3 as metadataDevice for device /dev/sda and let ceph-volume lvm batch decide how to create volumes
2020-11-28 14:31:17.695991 I | cephosd: configuring new device sdb
2020-11-28 14:31:17.695996 I | cephosd: using md0p3 as metadataDevice for device /dev/sdb and let ceph-volume lvm batch decide how to create volumes
2020-11-28 14:31:17.696010 D | exec: Running command: stdbuf -oL ceph-volume --log-path /tmp/ceph-log lvm batch --prepare --bluestore --yes --osds-per-device 1 /dev/sda /dev/sdb --db-devices /dev/md0p3 --report
2020-11-28 14:31:18.791881 D | exec: Traceback (most recent call last):
2020-11-28 14:31:18.791940 D | exec:   File "/usr/sbin/ceph-volume", line 11, in <module>
2020-11-28 14:31:18.791945 D | exec:     load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()
2020-11-28 14:31:18.791950 D | exec:   File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 40, in __init__
2020-11-28 14:31:18.791954 D | exec:     self.main(self.argv)
2020-11-28 14:31:18.791957 D | exec:   File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc
2020-11-28 14:31:18.791960 D | exec:     return f(*a, **kw)
2020-11-28 14:31:18.791964 D | exec:   File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 151, in main
2020-11-28 14:31:18.791967 D | exec:     terminal.dispatch(self.mapper, subcommand_args)
2020-11-28 14:31:18.791971 D | exec:   File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
2020-11-28 14:31:18.791974 D | exec:     instance.main()
2020-11-28 14:31:18.791977 D | exec:   File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 42, in main
2020-11-28 14:31:18.791981 D | exec:     terminal.dispatch(self.mapper, self.argv)
2020-11-28 14:31:18.791984 D | exec:   File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
2020-11-28 14:31:18.791988 D | exec:     instance.main()
2020-11-28 14:31:18.791991 D | exec:   File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
2020-11-28 14:31:18.791994 D | exec:     return func(*a, **kw)
2020-11-28 14:31:18.791997 D | exec:   File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 322, in main
2020-11-28 14:31:18.792001 D | exec:     self._get_explicit_strategy()
2020-11-28 14:31:18.792004 D | exec:   File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 332, in _get_explicit_strategy
2020-11-28 14:31:18.792008 D | exec:     self._filter_devices()
2020-11-28 14:31:18.792011 D | exec:   File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 385, in _filter_devices
2020-11-28 14:31:18.792014 D | exec:     raise RuntimeError(err.format(len(devs) - len(usable)))
2020-11-28 14:31:18.792018 D | exec: RuntimeError: 1 devices were filtered in non-interactive mode, bailing out
failed to configure devices: failed to initialize devices: failed ceph-volume report: exit status 1

Even tried to patch daemon/ceph/osd/volume.go:

2020-11-28 13:23:18.693154 D | exec: Running command: stdbuf -oL ceph-volume --log-path /tmp/ceph-log lvm batch --prepare --bluestore --osds-per-device 1 /dev/sda /dev/sdb --db-devices /dev/md0p3 --report --format json
2020-11-28 13:23:19.667110 D | cephosd: ceph-volume report: {
    "changed": true,
    "osds": [
        {
            "block.db": {},
            "data": {
                "human_readable_size": "7.28 TB",
                "parts": 1,
                "path": "/dev/sda",
                "percentage": 100.0,
                "size": 7451
            }
        },
        {
            "block.db": {},
            "data": {
                "human_readable_size": "7.28 TB",
                "parts": 1,
                "path": "/dev/sdb",
                "percentage": 100.0,
                "size": 7451
            }
        }
    ],
    "vgs": []
}
failed to configure devices: failed to initialize devices: ceph-volume did not use the expected metadataDevice [md0p3]

ceph_version: "v15.2.6"
rook_version: "v1.5.1"
@chrono2002 chrono2002 added the bug label Nov 28, 2020
@satoru-takeuchi
Copy link
Member

Is it possible to use raid here? Or there is another reliable solution?

No. Rook doesn't accept mdraid. Rook use replication and erasure coding to avoid data loss.

@chrono2002
Copy link
Author

Is it possible to use raid here? Or there is another reliable solution?

No. Rook doesn't accept mdraid. Rook use replication and erasure coding to avoid data loss.

the same is for metadata device?

@satoru-takeuchi
Copy link
Member

the same is for metadata device?

Yes.

@chrono2002
Copy link
Author

chrono2002 commented Nov 30, 2020 •
edited

looks like ceph doesn't accept partitions
ceph/ceph-ansible#4577
when I changed metadataDevice to /dev/md1 it goes further but bumped at:
https://tracker.ceph.com/issues/40776
https://158.69.68.89/issues/47831
so if you want to use md raid device you need to stay v14.2.7

@github-actions
Copy link

github-actions bot commented Mar 1, 2021

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

@github-actions
Copy link

github-actions bot commented Mar 8, 2021

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

@github-actions github-actions bot closed this as completed Mar 8, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug wontfix
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@satoru-takeuchi @chrono2002