Skip to content

Commit

Permalink
file/process encodings are automatically detected. This whould fix is…
Browse files Browse the repository at this point in the history 
…sue #72
  • Loading branch information
@ralequi
ralequi committed May 31, 2023
1 parent 6176dd7 commit dccbc1e
Show file tree
Hide file tree
Showing 6 changed files with 145 additions and 117 deletions.
2 changes: 1 addition & 1 deletion CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ Version 1.2.5
- Minnor fix on serial NVME readings (MR [#69](https://github.com/truenas/py-SMART/pull/69)). Thanks @Heidistein
- Minnor fix on reads/writes NVME readings (MR [#71](https://github.com/truenas/py-SMART/pull/71). Thanks @Heidistein
- Fixed capacity detection in some devices & envirorments (issue [#72](https://github.com/truenas/py-SMART/issues/72)). Thanks @Shablykinm

- Now file/process encodings are automatically detected. This whould fix issue [#72](https://github.com/truenas/py-SMART/issues/72)).

Version 1.2.4
=============
Expand Down
33 changes: 31 additions & 2 deletions pySMART/smartctl.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
from .utils import SMARTCTL_PATH
from typing import List, Tuple, Union, Optional

import chardet
import logging
import os

Expand Down Expand Up @@ -145,9 +146,37 @@ def _exec(self, cmd: List[str]) -> Tuple[List[str], int]:
"""
proc = Popen(cmd, stdout=PIPE, stderr=PIPE)

_stdout, _stderr = [i.decode('utf8') for i in proc.communicate()]
_stdout, _stderr = [i for i in proc.communicate()]

return _stdout.split('\n'), proc.returncode
return self._decode_output(_stdout), proc.returncode

def _decode_output(self, raw_output: bytes) -> List[str]:
""" Decodes the raw output from smartctl
This, should detect the encoding and decode the output
Args:
raw_output (bytes): The raw output from smartctl
Returns:
List[str]: A raw line-by-line output from smartctl
"""
# Detect the encoding
encoding: Optional[str] = None
try:
encoding = chardet.detect(raw_output)['encoding']
except:
pass

if not encoding:
encoding = 'utf-8'

if encoding not in ['utf-8', 'ascii']:
logger.warning(f"Detected encoding: {encoding}")

# Decode the output
decoded_output = raw_output.decode(encoding).splitlines()

return decoded_output

def scan(self) -> List[str]:
"""Queries smartctl with option --scan-open
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ classifiers = [
'Topic :: Software Development :: Libraries',
'Topic :: Software Development :: Libraries :: Python Modules',
]
dependencies = ['humanfriendly']
dependencies = ["chardet", 'humanfriendly']
dynamic = ["version"]


Expand Down
205 changes: 103 additions & 102 deletions tests/dataset/singletests/sata_ssd_1_issue_72/_-d_ata_--all__dev_sda
View file
Original file line number Diff line number Diff line change
@@ -1,102 +1,103 @@
smartctl 7.3 2022-02-28 r5338 [x86_64-w64-mingw32-w10-b19045] (sf-7.3-1)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Apacer AS340 SSDs
Device Model: Apacer AS340 240GB
Serial Number: LKC125R001176
LU WWN Device Id: 5 02b2a2 01d1c1b1a
Add. Product Id: mavlsata
Firmware Version: V4.25.0
User Capacity: 240 057 409 536 bytes [240 GB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
TRIM Command: Available
Device is: In smartctl database 7.3/5319
ATA Version is: ACS-3 T13/2161-D revision 4
SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Fri May 26 11:16:44 2023 RTZ
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x02) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 0) seconds.
Offline data collection
capabilities: (0x75) SMART execute Offline immediate.
No Auto Offline data collection support.
Abort Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0002) Does not save SMART data before
entering power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x00) Error logging NOT supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 90) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.

SMART Attributes Data Structure revision number: 5
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
9 Power_On_Hours 0x0033 100 100 010 Pre-fail Always - 3517
12 Power_Cycle_Count 0x0033 100 100 010 Pre-fail Always - 7
161 Unknown_Attribute 0x0033 100 100 010 Pre-fail Always - 0
164 Average_Erase_Count 0x0033 100 100 010 Pre-fail Always - 66135
165 Unknown_Attribute 0x0033 100 100 010 Pre-fail Always - 87
166 Later_Bad_Block_Count 0x0033 100 100 010 Pre-fail Always - 3
167 SSD_Protect_Mode 0x0033 100 100 010 Pre-fail Always - 64
169 Unknown_Attribute 0x0033 100 100 010 Pre-fail Always - 0
176 Erase_Fail_Count_Chip 0x0033 100 100 010 Pre-fail Always - 0
177 Wear_Leveling_Count 0x0033 100 100 010 Pre-fail Always - 0
178 Used_Rsvd_Blk_Cnt_Chip 0x0033 100 100 010 Pre-fail Always - 0
192 Unexpect_Power_Loss_Ct 0x0033 100 100 010 Pre-fail Always - 0
194 Temperature_Celsius 0x0033 100 100 010 Pre-fail Always - 47
195 Hardware_ECC_Recovered 0x0033 100 100 010 Pre-fail Always - 0
199 UDMA_CRC_Error_Count 0x0033 100 100 010 Pre-fail Always - 0
241 Total_LBAs_Written 0x0033 100 100 010 Pre-fail Always - 3848
242 Total_LBAs_Read 0x0033 100 100 010 Pre-fail Always - 1268
243 Unknown_Attribute 0x0033 100 100 010 Pre-fail Always - 0
244 Unknown_Attribute 0x0033 100 100 010 Pre-fail Always - 0
245 Unknown_Attribute 0x0033 100 100 010 Pre-fail Always - 0
246 Unknown_Attribute 0x0033 100 100 010 Pre-fail Always - 0
248 Unknown_Attribute 0x0033 100 100 010 Pre-fail Always - 0
249 Unknown_Attribute 0x0033 100 100 010 Pre-fail Always - 0
250 Read_Error_Retry_Rate 0x0033 100 100 010 Pre-fail Always - 0
251 Unknown_Attribute 0x0033 100 100 010 Pre-fail Always - 0
252 Unknown_Attribute 0x0033 100 100 010 Pre-fail Always - 0
253 Unknown_Attribute 0x0033 100 100 010 Pre-fail Always - 0
254 Unknown_SSD_Attribute 0x0033 100 100 010 Pre-fail Always - 0

SMART Error Log not supported

SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
smartctl 7.3 2022-02-28 r5338 [x86_64-w64-mingw32-w10-b19045] (sf-7.3-1)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Apacer AS340 SSDs
Device Model: Apacer AS340 240GB
Serial Number: LKC125R001176
LU WWN Device Id: 5 02b2a2 01d1c1b1a
Add. Product Id: mavlsata
Firmware Version: V4.25.0
User Capacity: 240�057�409�536 bytes [240 GB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
TRIM Command: Available
Device is: In smartctl database 7.3/5319
ATA Version is: ACS-3 T13/2161-D revision 4
SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Fri May 26 13:33:02 2023 RTZ
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status: (0x02) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 0) seconds.
Offline data collection
capabilities: (0x75) SMART execute Offline immediate.
No Auto Offline data collection support.
Abort Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0002) Does not save SMART data before
entering power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x00) Error logging NOT supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 90) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.

SMART Attributes Data Structure revision number: 5
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
9 Power_On_Hours 0x0033 100 100 010 Pre-fail Always - 3519
12 Power_Cycle_Count 0x0033 100 100 010 Pre-fail Always - 7
161 Unknown_Attribute 0x0033 100 100 010 Pre-fail Always - 0
164 Average_Erase_Count 0x0033 100 100 010 Pre-fail Always - 66179
165 Unknown_Attribute 0x0033 100 100 010 Pre-fail Always - 87
166 Later_Bad_Block_Count 0x0033 100 100 010 Pre-fail Always - 3
167 SSD_Protect_Mode 0x0033 100 100 010 Pre-fail Always - 64
169 Unknown_Attribute 0x0033 100 100 010 Pre-fail Always - 0
176 Erase_Fail_Count_Chip 0x0033 100 100 010 Pre-fail Always - 0
177 Wear_Leveling_Count 0x0033 100 100 010 Pre-fail Always - 0
178 Used_Rsvd_Blk_Cnt_Chip 0x0033 100 100 010 Pre-fail Always - 0
192 Unexpect_Power_Loss_Ct 0x0033 100 100 010 Pre-fail Always - 0
194 Temperature_Celsius 0x0033 100 100 010 Pre-fail Always - 47
195 Hardware_ECC_Recovered 0x0033 100 100 010 Pre-fail Always - 0
199 UDMA_CRC_Error_Count 0x0033 100 100 010 Pre-fail Always - 0
241 Total_LBAs_Written 0x0033 100 100 010 Pre-fail Always - 3851
242 Total_LBAs_Read 0x0033 100 100 010 Pre-fail Always - 1268
243 Unknown_Attribute 0x0033 100 100 010 Pre-fail Always - 0
244 Unknown_Attribute 0x0033 100 100 010 Pre-fail Always - 0
245 Unknown_Attribute 0x0033 100 100 010 Pre-fail Always - 0
246 Unknown_Attribute 0x0033 100 100 010 Pre-fail Always - 0
248 Unknown_Attribute 0x0033 100 100 010 Pre-fail Always - 0
249 Unknown_Attribute 0x0033 100 100 010 Pre-fail Always - 0
250 Read_Error_Retry_Rate 0x0033 100 100 010 Pre-fail Always - 0
251 Unknown_Attribute 0x0033 100 100 010 Pre-fail Always - 0
252 Unknown_Attribute 0x0033 100 100 010 Pre-fail Always - 0
253 Unknown_Attribute 0x0033 100 100 010 Pre-fail Always - 0
254 Unknown_SSD_Attribute 0x0033 100 100 010 Pre-fail Always - 0

SMART Error Log not supported

SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

12 changes: 6 additions & 6 deletions tests/dataset/singletests/sata_ssd_1_issue_72/device.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,8 @@
"type": "Pre-fail",
"updated": "Always",
"when_failed": "-",
"raw": "3517",
"raw_int": 3517,
"raw": "3519",
"raw_int": 3519,
"thresh": 10,
"value": "100",
"value_int": 100,
Expand Down Expand Up @@ -257,8 +257,8 @@
"type": "Pre-fail",
"updated": "Always",
"when_failed": "-",
"raw": "66135",
"raw_int": 66135,
"raw": "66179",
"raw_int": 66179,
"thresh": 10,
"value": "100",
"value_int": 100,
Expand Down Expand Up @@ -538,8 +538,8 @@
"type": "Pre-fail",
"updated": "Always",
"when_failed": "-",
"raw": "3848",
"raw_int": 3848,
"raw": "3851",
"raw_int": 3851,
"thresh": 10,
"value": "100",
"value_int": 100,
Expand Down
8 changes: 3 additions & 5 deletions tests/smartctlfile.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -59,11 +59,9 @@ def generic_call(self, params: List[str], pass_options=False) -> Tuple[List[str]
logger.trace("Opening file: {0}".format(filename))

try:
with open(filename, encoding='utf-8') as f:
raw_data = f.readlines()
with open(filename, mode='rb') as f:
raw_data = f.read()
except:
raise SmartctlfileSampleNotFound(filename, final_params)

_stdout = [i.split('\n')[0] for i in raw_data]

return _stdout, 0
return self._decode_output(raw_data), 0

0 comments on commit dccbc1e

Please sign in to comment.