Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Help needed. HPA/DCO [Completed] #444

Closed
PartialVolume opened this issue Mar 3, 2023 · 76 comments
Closed

Help needed. HPA/DCO [Completed] #444

PartialVolume opened this issue Mar 3, 2023 · 76 comments

Comments

@PartialVolume
Copy link
Collaborator

PartialVolume commented Mar 3, 2023 •
edited
Loading

I'm currently working on the HPA/DCO code and have come across an issue with hdparm incorrectly parsing the sense data when it tries to calculate the real max sectors, this bug exists in v9.60 and has since been fixed in v9.65 however I don't know exactly which version it was fixed in, irrespective nwipe will need to cope with this bug if it's run on a LTS distro in particular that's running older versions of hdparm.

So I plan on parsing the sense data myself when the real max sectors don't make any sense.

However the largest drive I have for testing is 2TB and I need some sense data from a 10TB or greater drive to make sure I decode the size correctly.

If somebody could run the following command on a drive that is larger than 10TB and post the output here that would be appreciated.

The command is sudo hdparm --verbose --dco-identify /dev/sdxxxx

Thanks

either reply here or over in ShredOS discussions discussions-Help-Needed
@mdcato
Copy link

mdcato commented Mar 3, 2023 via email

@PartialVolume
Copy link
Collaborator Author

@mdcato excellent, the version isn't important, however if you could let me know what version it is just for reference, it's really just the hex sense data I'm interested in that's spouted out by using the --verbose option, it will mostly be zeros except for the first line and a couple of non zero bytes at the end.

If you could post the whole output though including the real max sector info at the end.

Thanks 👍

@mdcato
Copy link

mdcato commented Mar 3, 2023 via email

@PartialVolume
Copy link
Collaborator Author

Just checking but were both drives connected via SATA and not via USB?

Unfortunately that data hasn't provided the answer I was looking for, there is a couple of "SG_IO: bad/missing sense data" errors and the hex data doesn't contain anything other than zeros so hdparm is failing to get the data.

After the ninth byte after incoming data: I was expecting to see a hex value that represented the max sectors. The real max sectors is also reported a '1' should be about 19531250000 sectors for 10TB drive. In the sense data there should be a string much like 0x5039278c4

Here's an example of what it should look like, the highlighted hex value 0xaf88e0e8 which is 2000398933504 bytes (2.0TB)

outgoing cdb: 85 08 0e 00 c2 00 01 00 00 00 00 00 00 40 b1 00
SG_IO: ATA_16 status=0x0, host_status=0x0, driver_status=0x0
SG_IO: sb[]: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
incoming_data: 02 00 00 00 7f 00 af 88 e0 e8 00 00 00 00 88 00 15 00

The bad or missing sense data could be because of the interface being used or maybe these drives don't conform which I think is unlikely.

@mdcato
Copy link

mdcato commented Mar 4, 2023 via email

@PartialVolume
Copy link
Collaborator Author

PartialVolume commented Mar 4, 2023 •
edited
Loading

Very odd, maybe something in the bios that's messing with access to the SATA bus?

What's the output of lspci, is there a clue as to the chipset being used for SATA?

Would be interesting to see whether this USB unitek USB to SATA adapter works where the motherboards SATA doesn't! This adapter works perfectly for me and uses the same SATA chip used in some laptops.

Below is the dco-identify output from hdparm v9.65 having parsed the sense data correctly and showing the correct real max sectors and no SG_IO: bad/missing sense data errors, on a 2TB Western Digital Green WDC WD20EURS-63SPKY0. Also works on a Dell Optiplex 9010.

sudo ./hdparm --verbose --dco-identify /dev/sd
/dev/sda:
APT: USB ID = 0x174c:0x55aa (0x100)
APT: Unsupported device
outgoing cdb: 85 08 0e 00 c2 00 01 00 00 00 00 00 00 40 b1 00
SG_IO: ATA_16 status=0x0, host_status=0x0, driver_status=0x0
SG_IO: sb[]: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
incoming_data: 02 00 00 00 7f 00 af 88 e0 e8 00 00 00 00 88 00 15 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 a5 3e
SG_IO: desc[]: 00 00
ATA_16 stat=00 err=00 nsect=00 lbal=00 lbam=00 lbah=00 dev=00
DCO Checksum verified.
DCO Revision: 0x0002
The following features can be selectively disabled via DCO:
Transfer modes:
udma0 udma1 udma2 udma3 udma4 udma5 udma6
Real max sectors: 3907029168
ATA command/feature sets:
security HPA
SATA command/feature sets:
NCQ interface_power_management SSP

@PartialVolume
Copy link
Collaborator Author

@mdcato Is this your bios?, anything in the SATA configuration that could be causing it?

Aggresive LPM if enabled, maybe the drive is in lower power state and hdparm isn't hanging around to see if it wakes up? but then on the second attempt it should work.

https://youtu.be/5PYroA_whjU?t=411

Screenshot_20230304_172602

@PartialVolume
Copy link
Collaborator Author

Decoding the sense response with the command sg_decode_sense 70 00 05 00 00 00 00 0a 04 51 40 01 21 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

produces the response

Fixed format, current; Sense key: Illegal Request
Additional sense: Unaligned write command

Exit code 5 : the device reports an "illegal request" with an additional sense code other than "invalid operation code". This is often a supported command with a field set requesting an unsupported capability. For commands that require a "service action" field (e.g. READ CAPACITY(16) ) this value can indicate that the command is not supported

So if that is decoded correctly it's saying the command being issued is an illegal request. Doesn't help much does seem to point to something blocking access much like you would get with a USB adapter that doesn't support the full ATA specification. I don't know whether it's the drive, controller or bios but like you say, try those discs on a different motherboard and if they work at least you know it's nothing to do with the discs.

@mdcato
Copy link

mdcato commented Mar 5, 2023 via email

@PartialVolume
Copy link
Collaborator Author

It's a puzzle, how about sudo hdparm -I /dev/sda. Just found a 3TB Western Digital. Do you get all this info when you just run the -I option

/dev/sda:

ATA device, with non-removable media
        Model Number:       WDC WD30EURX-63T0FY0                    
        Serial Number:      WD-XXXXXXXXXXXXXX
        Firmware Revision:  80.00A80
        Transport:          Serial, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0
Standards:
        Supported: 9 8 7 6 5 
        Likely used: 9
Configuration:
        Logical         max     current
        cylinders       16383   16383
        heads           16      16
        sectors/track   63      63
        --
        CHS current addressable sectors:    16514064
        LBA    user addressable sectors:   268435455
        LBA48  user addressable sectors:  5860533168
        Logical  Sector size:                   512 bytes
        Physical Sector size:                  4096 bytes
        Logical Sector-0 offset:                  0 bytes
        device size with M = 1024*1024:     2861588 MBytes
        device size with M = 1000*1000:     3000592 MBytes (3000 GB)
        cache/buffer size  = unknown
        Nominal Media Rotation Rate: 5400
Capabilities:
        LBA, IORDY(can be disabled)
        Queue depth: 32
        Standby timer values: spec'd by Standard, with device specific minimum
        R/W multiple sector transfer: Max = 16  Current = 0
        DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6 
             Cycle time: min=120ns recommended=120ns
        PIO: pio0 pio1 pio2 pio3 pio4 
             Cycle time: no flow control=120ns  IORDY flow control=120ns
Commands/features:
        Enabled Supported:
           *    SMART feature set
                Security Mode feature set
           *    Power Management feature set
           *    Write cache
           *    Look-ahead
           *    Host Protected Area feature set
           *    WRITE_BUFFER command
           *    READ_BUFFER command
           *    NOP cmd
           *    DOWNLOAD_MICROCODE
                Power-Up In Standby feature set
           *    SET_FEATURES required to spinup after power up
                SET_MAX security extension
           *    48-bit Address feature set
           *    Device Configuration Overlay feature set
           *    Mandatory FLUSH_CACHE
           *    FLUSH_CACHE_EXT
           *    SMART error logging
           *    SMART self-test
                Media Card Pass-Through
           *    General Purpose Logging feature set
           *    64-bit World wide name
           *    URG for READ_STREAM[_DMA]_EXT
           *    URG for WRITE_STREAM[_DMA]_EXT
           *    IDLE_IMMEDIATE with UNLOAD
           *    WRITE_UNCORRECTABLE_EXT command
           *    {READ,WRITE}_DMA_EXT_GPL commands
           *    Segmented DOWNLOAD_MICROCODE
           *    Gen1 signaling speed (1.5Gb/s)
           *    Gen2 signaling speed (3.0Gb/s)
           *    Gen3 signaling speed (6.0Gb/s)
           *    Native Command Queueing (NCQ)
           *    Host-initiated interface power management
           *    Phy event counters
           *    Idle-Unload when NCQ is active
           *    NCQ priority information
           *    READ_LOG_DMA_EXT equivalent to READ_LOG_EXT
                DMA Setup Auto-Activate optimization
                Device-initiated interface power management
           *    Software settings preservation
           *    SMART Command Transport (SCT) feature set
           *    SCT Write Same (AC2)
           *    SCT Features Control (AC4)
           *    SCT Data Tables (AC5)
                unknown 206[12] (vendor specific)
                unknown 206[13] (vendor specific)
                unknown 206[14] (vendor specific)
Security: 
        Master password revision code = 65534
                supported
        not     enabled
        not     locked
        not     frozen
        not     expired: security count
                supported: enhanced erase
        442min for SECURITY ERASE UNIT. 442min for ENHANCED SECURITY ERASE UNIT.
Logical Unit WWN Device Identifier: 50014ee25f01493d
        NAA             : 5
        IEEE OUI        : 0014ee
        Unique ID       : 25f01493d
Checksum: correct

@mdcato
Copy link

mdcato commented Mar 5, 2023 via email

@PartialVolume
Copy link
Collaborator Author

One thing that I noticed was your drive is frozen. All my drives when I run this command are not frozen. I wonder if that's got something to do with it?

@mdcato
Copy link

mdcato commented Mar 5, 2023 via email

@PartialVolume
Copy link
Collaborator Author

I just had a read of the Seagate ST10000NM0016 product manual. It makes no mention of the host protected area (HPA) or device configuration overlay (DCO). Presumably it's been dropped? The maximum logical addressable sector is available using the 0xEC command which is what hdparm uses to display it's output for the sudo hdparm --verbose -N /dev/sdx. Unless something else is going on here, that command should work and it should display the max addressable sectors or at least if it doesn't if the block of incoming data is there I can decode it from that.

So if you could run sudo hdparm --verbose -N /dev/sdx and post the output.

Here's a section from the Seagate manual, word 100-103
Screenshot_20230305_180653

@PartialVolume
Copy link
Collaborator Author

According to ATA_Command_Matrix the 0xB1 device configuration overlay was dropped in ACS-2, I think we are at ACS-5 now.

Screenshot_20230305_182103

@PartialVolume
Copy link
Collaborator Author

All useful information as I expected HPA/DCO to exist in all drives up to the present but apparently not. I can bear that in mind when figuring out how I work out whether a HPA exists or not. I tell you, it's a mine field ! :-)

@PartialVolume
Copy link
Collaborator Author

PartialVolume commented Mar 5, 2023 •
edited
Loading

One thing I have realised about the existence of an HPA is if anybody is using hdparm and thinks that running hdparm -N /dev/sdx and it responds with the message "HPA is disabled" and that means there is no hidden area then they are being misled as the statement "HPA is disabled" may be completely inaccurate.
Screenshot_20230305_192816

By using the dco-setmax command to reduce the apparent size of the disc by say 512 bytes, it changes both the 1st and 2nd max sector sizes as displayed by sudo hdparm -N /dev/sdx and because both values match hdparm responds with HPA disabled. Yet I just created a hidden block where I stored some data, then hid it then showed it. So you cannot trust the message HPA disabled!, you have to reference the real max sectors value with the command sudo hdparm --dco-identify /dev/sdx and unless all three values match then there is a hidden area at the end of the disc.

Makes me wonder how many people get confused by that message 'HPA is disabled', without referencing the real max sectors.

@mdcato
Copy link

mdcato commented Mar 6, 2023 via email

@PartialVolume
Copy link
Collaborator Author

Thanks, that's useful. The message output by the 10TB drive, I've not come across before.

max sectors = 19532873728/19532873728, ACCESSIBLE MAX ADDRESS disabled

I'm surprised the 1TB doesn't work with --dco-identify but then maybe it's less than 4 or 5 years old?

@mdcato
Copy link

mdcato commented Mar 6, 2023 via email

@PartialVolume
Copy link
Collaborator Author

PartialVolume commented Mar 11, 2023 •
edited
Loading

@mdcato I'm getting very close to completing the two page PDF report that nwipe generates. I've tested on the drives I have which are all old and all support HPA/DCO. However I need to see how nwipe works with one of your drives that don't have HPA/DCO.

I don't know whether you have time for this but if you do, any chance you could download the latest master and compile it, then run it on say your 2TB to start with. The one that didn't work with the hdparm --dco-identify command.

What I'm interested in are two things to start with.

  • Start a wipe on the drive, pick what ever method you want, run it for a few seconds then abort it. PDF will be generated.
  • Start a wipe on the drive, let it complete the wipe. Press spacebar to exit and PDF will be automatically generated.

Start nwipe with the -q option if you want to anonymize the data.

At the end of each wipe a PDF report will be generated.

Can you post the PDF report and also the first few lines of the nwipe log that's displayed in the terminal, basically the drive enumeration detail as shown below:

[2023/03/11 22:38:23]    info: nwipe 0.34.5 Development code, not for production use!
[2023/03/11 22:38:23]    info: Linux version 5.19.0-32-generic (buildd@lcy02-am
                               d64-026) (x86_64-linux-gnu-gcc (Ubuntu 11.3.0-1
                               ubuntu1~22.04) 11.3.0, GNU ld (GNU Binutils for
                                Ubuntu) 2.38) #33~22.04.1-Ubuntu SMP PREEMPT_D
                               YNAMIC Mon Jan 30 17:03:34 UTC 2
[2023/03/11 22:38:23]  notice: Found /dev/sdb,  USB, ST350031 2CS, 512 KB, S/N=XXXXXXXXXXXXXXX
[2023/03/11 22:38:23]    info: HPA values 1000 / 976773168 on /dev/sdb
[2023/03/11 22:38:23]    info: hdparm:DCO Real max sectors reported as 976773168 on /dev/sdb
[2023/03/11 22:38:23]    info: NWipe: DCO Real max sectors reported as 976773168 on /dev/sdb
[2023/03/11 22:38:23] warning: HIDDEN AREA DETECTED! on /dev/sdb
[2023/03/11 22:38:23]    info: func:nwipe_read_dco_real_max_sectors(), DCO real max sectors = 976773168
[2023/03/11 22:38:23]    info:  
[2023/03/11 22:38:23]    info: bios-version = 1.07.02TNO

If it works as it's supposed to, bearing in mind this is the first test on a non HPA/DCO device, you should see a sea of green text and a green tick icon in the top right corner with no red text, unlike the PDF below where I created an enormous hidden area and only wiped the visible part of the disc.

Much appreciated.

PDF nwipe_report_2023-03-11-22-38-43_Model_ST350031_2CS_Serial_XXXXXXXXXXXXXXX.pdf

Screenshot_20230311_231026

Screenshot_20230311_224351

@mdcato
Copy link

mdcato commented Mar 11, 2023 via email

@PartialVolume
Copy link
Collaborator Author

Zero will be fine, I wouldn't worry about verification unless you really want to.

Also can you check the GUI it should be showing this, alternating with the drive model/serial no.

Screenshot_20230311_232311

@mdcato
Copy link

mdcato commented Mar 12, 2023 via email

@PartialVolume
Copy link
Collaborator Author

@mdcato not sure what happened there, there's no screenshots or files, either in the GitHub pages or the email.

@mdcato
Copy link

mdcato commented Mar 12, 2023 via email

@mdcato
Copy link

mdcato commented Mar 12, 2023 via email

@PartialVolume
Copy link
Collaborator Author

PDF and snapshots still not coming through. The log has given me enough info to know what I need to change. I'll make the changes over the next few days and let you know when it's ready for testing again. Thanks for helping out with this. Saves me having to buy a 10TB drive !

@PartialVolume
Copy link
Collaborator Author

PartialVolume commented Mar 14, 2023 •
edited
Loading

@mdcato

Another potential anomaly on the GUI is the two Enterprise drives' temperature is always red and flashing. They do run hot, both 50C at the end of the wipe, but Seagate manual says operating temperature is 5-60C.

I've committed some code to the master that logs the four temperatures that are obtained from the drive via the hwmon module. See PR for details

If you download and compile this and run nwipe, no need to start any wipes or select any drives, just immediately CNTRL C to abort and you'll see at the bottom of the log the high critical, highest, lowest and low critical temperatures. I have discovered one anomaly with the code. Basically to do with what I thought highest and lowest meant.

I'm sure I read in the HWMON docs that highest and lowest were the continuous running temperatures, but that's not what they are on this Seagate ST3500312CS. They change value and as they actually say are the highest temperature reached by the drive and the lowest. That means highest may start of at room temperature and then may exceed critical, but as the drive temperature drops highest stays at the highest temperature it reached. I think it's probably reset after a power cycle.

However what this could mean is that your drives, even a few seconds, reached critical, i.e 65 deg.C then so would highest and it would stay there. If the drive then cooled down, the red flashing temperature might change to solid red but it would never drop to white on blue text, it would stay at red on blue even if the drive temperature dropped to 20 deg.C

Does that behaviour ring any bells. It's easy enough to fix, I just ignore the highest and lowest values and just make the temperature flash when it reaches critical high or low.

Would be nice to see what your drives say their critical temperature limits are reported as in the nwipe log and if that matches the 65 deg.C specified in the data sheet.

@PartialVolume
Copy link
Collaborator Author

Just made the changes to the code, just uploading it now 0.34.6.

I've done away with the flashing red and just have non flashing text.

So if the drive temperature exceeds the high critical temperature the temperature in the GUI will change to red text on blue background.

If the drive temperature drops below the low critical value (if it exists) then the temperature changes to black text on blue background.

If the drive reports no critical temperatures, the temperature text defaults to white text on blue.

@PartialVolume
Copy link
Collaborator Author

PartialVolume commented Mar 16, 2023 •
edited
Loading

New code uploaded, I'm expecting the temperature code to not be red now on your discs with this new code as it shouldn't be getting confused by the zero values in critical and min/max.

Now validates temperatures rather than assuming the data is valid, checking for the following:

  • Checks for 0 in high critical and max.
  • Copes with missing critical or max min data or partially missing data.
  • Checks the high critical and max are the right way round
  • Checks the low critical and min are the right way round
  • Temperature is displayed in 5 different ways

Meaning of color changes

  • White text on blue - temperature with spec. (or no spec available)
  • Red text on blue - max temperature reached.
  • White text on red - critical high temperature reached.
  • Black text on blue - minimum temperature reached.
  • White text on black - critical low temperature reached.

Test code showing temperature cycling between +10 and -10 with critical at +8 and -8 and min/max at -5 & +5.
The colors of the temperature text and background change when those settings are reached. As this is a test, those values are low, just for test purposes, on a actual disc they are more like critical +65, max +60, min -2 critical min -5

showing_new_temperature_style-2023-03-15_23 26 27 mp4

@PartialVolume
Copy link
Collaborator Author

@.*** device]# find / -name "temp1_lowest" -exec cat {} ;
48000
39000
38000

Dividing those values by 1000 give us our degrees C. In that same directory there must be another file containing the actual realtime drive temperature.

@mdcato
Copy link

mdcato commented Mar 16, 2023 via email

@PartialVolume
Copy link
Collaborator Author

That's great, thanks at least we know the file naming convention is the same. I'm none the wiser as to whether it's just that that these drives just don't produce that sort of temperature limit data or whether hwmon is failing to retrieve it. I don't know whether smartctl -a displays any temperature related limits, I don't remember seeing anything.

Anyway, let me know how it goes with the new code. I'll be fixing the HPA GUI and report issues tomorrow, so more code to follow.

@PartialVolume
Copy link
Collaborator Author

Just been reading the data sheets for these Seagate Enterprise drives Vs the ATA/ATAPI command set ACS3 and from what I can make out the enterprise drives ( or maybe newer Seagate drives??) Don't support the upper/lower critical trip points or the min/max trip points. Just the actual temperature and the the lowest/highest. The drive can respond with an over temperature sense for various commands but you don't seem to be able to interrogate the drive for the trip value other than reading the product manual in which they specify 60 degree case temperature using a thermal sensing device.

Not to worry though, as it stands nwipe will just show the temperature in white text on blue if it can't determine a sensible trip point.

One other thing I noted. These drives return 0xFF in the actual temperature if the internal sensor has failed. So if you ever see either a temperature of 255C or -1C you'll know why.

@mdcato
Copy link

mdcato commented Mar 16, 2023 via email

@PartialVolume
Copy link
Collaborator Author

That's great, thanks. I wasn't even aware the revision wasn't in the log, I'd assumed as I see it in the log that's sent out to the console it was in the log file too. Adding that into tonight's update.

@mdcato
Copy link

mdcato commented Mar 16, 2023 via email

@PartialVolume
Copy link
Collaborator Author

New code uploaded and ready for testing :-) #457

@PartialVolume
Copy link
Collaborator Author

@mdcato I obtained a used 10TB EXOS ST10000NM001G-2MW103 drive (with issues) and was able to understand better how this drive differs from others. I then made changes to the code to accommodate the way HPA is determined for these drives. These drives don't support DCO but they do support HPA. I created a hidden sector in the HPA and nwipe now correctly identifies it as having a hidden sector. I'm currently running a single wipe, no blanking, no verification on this drive with a hidden sector. It's estimated time to completion is about 12hrs, I'm currently getting about 180MB/s on the unitek USB adapter.

The master contains the latest changes if you want to give it a try on your EXOS drive. All the fields in the PDF should be correctly displaying their data. Famous last words. :-) See #462

@mdcato
Copy link

mdcato commented Apr 5, 2023 via email

@mdcato
Copy link

mdcato commented Apr 5, 2023 via email

@PartialVolume
Copy link
Collaborator Author

Here’s a screenshot of the current run with the Remaining time at 28:35:43.

No snapshot coming through. I think you will have to paste it via the web page rather than email.

@PartialVolume
Copy link
Collaborator Author

PartialVolume commented Apr 5, 2023 •
edited
Loading

@mdcato Just to confirm the master version is v0.34.83

Thought I'd double check just to make sure the HPA was being read correctly, so I aborted the wipe and reset the HPA. Here's what you should see at the top of the log for a EXOS with and without a hidden sector.

EXOS without a hidden sector.

Note the two values match. max sectors = 19532873728/19532873728, accessible max address disabled

[2023/04/05 20:41:25] notice: Found /dev/sdb, USB, ST10000N M001G-2MW103, 10 TB, S/N=XXXXXXXXX
[2023/04/05 20:41:25] info: HPA: max sectors = 19532873728/19532873728, accessible max address disabled
on /dev/sdb
[2023/04/05 20:41:25] info: HPA values 19532873728 / 19532873728 on /dev/sdb
[2023/04/05 20:41:25] info: DCO Real max sectors not found
[2023/04/05 20:41:25] info: func:nwipe_read_dco_real_max_sectors(), DCO real max sectors = 0

EXOS with one hidden sector

Note the two values differ by one HPA: max sectors = 19532873727/19532873728, accessible max address enabled

[2023/04/05 16:07:06] notice: Found /dev/sdb, USB, ST10000N M001G-2MW103, 10 TB, S/N=XXXXXXXX
[2023/04/05 16:07:06] info: HPA: max sectors = 19532873727/19532873728, accessible max address enabled
on /dev/sdb
[2023/04/05 16:07:06] info: HPA values 19532873727 / 19532873728 on /dev/sdb
[2023/04/05 16:07:06] info: DCO Real max sectors not found
[2023/04/05 16:07:06] warning: *********************************
[2023/04/05 16:07:06] warning: *** HIDDEN SECTORS DETECTED ! *** on /dev/sdb
[2023/04/05 16:07:06] warning: *********************************
[2023/04/05 16:07:06] info: func:nwipe_read_dco_real_max_sectors(), DCO real max sectors = 0

@mdcato
Copy link

mdcato commented Apr 5, 2023

Here's the missing screenshot.
Screenshot from 2023-04-05 13-48-01

@mdcato
Copy link

mdcato commented Apr 5, 2023

Yes, looking at my currently running log, both max sectors match.

@mdcato
Copy link

mdcato commented Apr 5, 2023

If my math (and calculator) are correct, it looks like the Remaining Time is the sum of all the drives' remaining time, rather than the longest (i.e. the 10TB drive in this case).

@PartialVolume
Copy link
Collaborator Author

PartialVolume commented Apr 5, 2023 •
edited
Loading

@mdcato I just found a bug in the pdf creation, reports a hidden sector while simultaneously reporting 0 hidden sectors. This is with WD drive attached. Up to you but I'd probably abort your wipe, I'll probably fix that bug this evening and upload v0.34.84.

I can see why you are reading 28 hours and I'm reading about 13hrs. You are writing twice the data. You have the blanking pass switched on, so you are doing a fill with ones then a fill with zeros. I'm just doing a fill with zeros and no blanking pass.

@mdcato
Copy link

mdcato commented Apr 5, 2023 via email

@PartialVolume
Copy link
Collaborator Author

PartialVolume commented Apr 5, 2023 •
edited
Loading

I've fixed the bug, uploading the new code in the next 30 minutes. v0.34.84. This wipe is on two USB adapters, one is a good one (Unitek) and the other doesn't respond properly DCO/HPA (due to the chip) but despite this nwipe detects HPA/DCO correctly.

Screenshot_20230405_222826

I need to write a function to pull in those temperatures, HWMON doesn't work with USB so I'm going to have to pull the temperatures from the drives myself. I know that I can access 194 (temperature) via the smart data so I'll try and get that done next but taking a break for the rest of the week, so probably do that next week.

Let me know how it goes with the wipes and thanks for your help.

@PartialVolume
Copy link
Collaborator Author

@mdcato v0.34.84 #463 now available

@PartialVolume
Copy link
Collaborator Author

As a reminder to myself for next week's work. When doing a multi disc wipe and one or more discs complete their wipes and one or more are still wiping, you then decide to abort, the PDF's for the wipes that were completed show as status Aborted rather than erased. That needs to be fixed!.

@mdcato
Copy link

mdcato commented Apr 6, 2023

Here are the results from #462 for ST4000NM0033, ST4000VN008 and ST10000NM0016. The usual --verbose log, the report PDFs, and some screenshots at beginning of subsequently aborted run due to blanking pass still selected showing HPA/DCO on GUI, and at end of completed run.

Screenshot from 2023-04-05 13-06-10
nwipe-20230405.log
Screenshot from 2023-04-05 13-06-00

Screenshot from 2023-04-06 11-48-19

nwipe_report_2023-04-06-09-12-18_Model_ST10000NM0016_1T_Serial_ZA20SSAR.pdf
nwipe_report_2023-04-06-00-18-35_Model_ST4000VN008_2DR1_Serial_ZGY1DA5C.pdf
nwipe_report_2023-04-06-01-08-31_Model_ST4000NM0033_9ZM_Serial_Z1Z7NVF7.pdf

@PartialVolume
Copy link
Collaborator Author

@mdcato I seem to be missing the log files.

Are you able to do that test with #463 as my test on the 10TB worked ok with and without hidden sectors using that latest version. Although mine is connected via USB. Shouldn't make any difference though.

If you get the same results with #463 , i.e incorrect hidden sectors status and crazy large number of hidden sectors then I'll take another look at the code next week and hook my 10TB directly to SATA rather than USB.

@mdcato
Copy link

mdcato commented Apr 6, 2023 via email

@PartialVolume
Copy link
Collaborator Author

The logfile link ended up between screenshots
Thanks, missed that the first time I looked at it.👍

@mdcato
Copy link

mdcato commented Apr 7, 2023

PartialVolume,
Here are the results of running #463 0.34.84. The PDF Disk Erasure Reports show a reasonable Size(Apparent), but the Size(Real) reports 512 bytes for all three drives. The Bytes Erased field has a LARGE percentage value, and the HPA/DCO Size also has a LARGE number of sectors.

nwipe-20230406.log
Screenshot from 2023-04-07 14-46-23
nwipe_report_2023-04-07-03-10-04_Model_ST4000VN008_2DR1_Serial_ZGY1DA5C.pdf
nwipe_report_2023-04-07-03-58-52_Model_ST4000NM0033_9ZM_Serial_Z1Z7NVF7.pdf
nwipe_report_2023-04-07-13-05-20_Model_ST10000NM0016_1T_Serial_ZA20SSAR.pdf

@PartialVolume
Copy link
Collaborator Author

@mdcato thanks, that's very useful, I can see from the logs that the DCO-Identify request is failing with a value of 1 sector, hdparm returns 1 on failure, I don't know why, it should really return 0 or -1. But anyway the code is incorrectly accepting a valid DCO greater than 0 so it incorrectly thinks the real size is one sector, from then on everything is calculated incorrectly. So I have a couple of options, only accept a valid DCO as >1 sector but also harden the code to check the sense data for an illegal request which I believe the drive issues then take the appropriate action. It's reading the HPA sector sizes correctly so should have reverted to using those when the DCO came back invalid.

I'll hopefully work on this next week, run some tests on my 10TB hooked up to the SATA as well as USB then let you re-run your tests. Hopefully I'll nail it this time.

@PartialVolume PartialVolume changed the title Help needed. Help needed. HPA/DCO [Completed] Nov 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@PartialVolume @mdcato