PFS3 partition validity issues when using >4GB templates

[tomarkus]

Hi Henrik,

I would like to report one issue I've stumbled upon when using HSTWB which is not related directly to your packet, but rather to PFS3 known and not fully fixed bug described here:
http://eab.abime.net/showpost.php?p=1110391&postcount=84

Here are reproduction steps:

Environment:

• HstWB version: stable v1.1.0
• Win10 Home, Version 1709, OS Build 16299.192, 64-bit,
• WinUAE 3.5.0, 64-bit.

Reproduction Steps:

1. Download 8GB, 16GB, 32GB, 64 GB HDF template from http://hstwb.firstrealize.com/,
2. Follow YT installation guide - https://www.youtube.com/watch?v=oppoOPUdmsA:

• Chose the same Amiga HW config template as in video,
• Choose any of ClassicWB version,
• Check all four EAB WHDLoad menus.

3. Run installed AmigaOS in WinUAE and copy progress.gadget file from DH0:MyFiles/LargeHD/128GB_Support/PFS3/classes/gadgets/ to DH0:Classes/Gadgets/.
4. Run PFSDoctor v1.6 from System:MyFiles/LargeHD/128GB_Support/PFS3/tools.
5. Select Check->Volume->Work and then press START button confirming all the dialogs.

Expected result:
There are no:

• corrupted directory entries,
• reserved, but not allocated blocks,
• anode related errors.

Actual Result:
There 200-300+ errors on a Work partition filled only with WHDLoad directories and games/demos images.

Additional information:

• This issue is not reproducible when using 4GB HDF template and 3300-3500MB Work partition.
• If we decide to keep using >4GB PFS partition, and then copy WHDLoad_Packs Demos, Games (ECS, AGA) to Work partition from ftp://grandis.nu/Commodore_Amiga/WHDLoad_Packs/ (especially games from Games_WHDLoad_UnpackOnAmiga/), we end up having FS3 Wrong Index Block ID / device not ready errors.

Considering above, I would suggest either switch to SFS or re-create >4GB HDF templates using multiple <4GB PFS3 partitions just to be safe and lower the risk of having nasty FS issues.

Best regards,
Tomasz

[henrikstengaard]

This is quite interesting. I will try and reproduce the issue with both ClassicWB and HstWB

[tomarkus]

I have contacted Toni as well - he asked me to conduct some additional tests with different partition sizes - <4GB, ~5GB and >6GB (PFS3-AIO is handling >6GB partition differently) and experimental PFSDoctor supporting PFS3-AIO modes. Hopefully, I will test everything today evening.

[henrikstengaard]

Just for information, all hdf files are prepared with WinUAE using hdtoolbox and uaehf.device. I will also try testing it this evening

[henrikstengaard]

And formatted with pfsformat from pfs53_3

[henrikstengaard]

I already had an image installed and build from the 16gb hdf template and running pfsdoctor reported 600+ errors, so that's a problem.

The method I used to partition and format the hdf was through Workbench 3.1 with HDToolBox that presumable comes from Workbench 3.1 as well. The process and files used are described here https://16bitdust.wordpress.com/2015/10/13/partitioning-16-gb-compact-flash-card-with-winuae-and-pfs3/.

When I read more about large harddisks and PFS3, it seems like it's best to partition and format a 4GB+ harddisk using Amiga OS 3.9 as it has a newer HDToolBox that supports and displays 4GB+ sizes correctly.

So for testing I have used Amiga OS 3.9 to partition and format a new 16gb hdf. I have also switched to PDS\3 dostype as it seems a better option for large harddisks. Then to it's installing ClassicWB/HstWB + WHDLoad games and run pfsdoctor afterwards to verify, if it fixes the errors.

[tomarkus]

Yeah, I was curious why you have decided to stick with original WB3.1 HDToolBox instead of HDInstTools:
http://aminet.net/package/disk/misc/hdinst

4GB support can be challenging: http://eab.abime.net/showthread.php?t=32256

I'm afraid that your proposal won't solve the problem fully - first of all PFSDoctor from PFS3-AIO is not fully supporting all FS modes. I think that is why I have received some experimental PFSDoctor version from Toni for testing purposes. Another thing is that there are multiple sources indicating that a PFS3-AIO v2.3 fix:

• Fixed very old PFS3 bug that caused Wrong Index Block ID error message. Only affected partitions larger than 5G.

is still there. I will try to find an exact scenario which leads to this error. Nevertheless, I think your approach can for sure help. Would you be able to provide me a new 8-16GB template with the changes? I will be happy to test them out as well.

[henrikstengaard]

HDToolBox vs HDInstTools... I always thought HDToolBox was the better choice. I assume HDToolBox in Amiga OS 3.9 will handle it correctly. Maybe HDInstTools is better, I'm still learning after actively configuring and using Amiga Workbench for years.

I ran install mode using latest v1.2.0-beta2 with the new 16gb hdf file and installed: HstWB, BetterWB, EAB WHDLoad Demos and Games AGA Menu's. After installation I ran both pfsdoctor 1.6 and pfsdoctor experimental and have attached the following screenshots and log file:


pfsdoctor_exp_check.log

Unfortunately there's still errors, but all of them are related to EmeraldMinesCD, which makes me want to investigate EmeraldMinesCD archive from UnpackOnAmiga further. All in all this seems good, but needs to be further investigated and retry without UnpackOnAmiga games.

Here's the new 16gb hdf file partitioned and formatted from Amiga OS 3.9 using default settings and pfs3-aio v2.3:
16gb_hdf_rdb_dh0-1000mb-pds3_dh1-13gb-pds3.zip

[henrikstengaard]

Just but the new image file in the images directory for hstwb installer and restart setup. Then it should be available when creating new images from a template.

[tomarkus]

Would you be able to provide me your email address? I would like to CC you into my conversation with Toni. It doesn't make any sense to have to separate threads about this - here is my mail: janiszewski.tomasz@gmail.com.

[henrikstengaard]

Sure, I have sent a mail to you.

[henrikstengaard]

I retried installing the new 16gb hdf file from scratch this time without UnpackOnAmiga games and it looks really good.

I did the following:

1. Computer has Windows 10 64-bit and WinUAE v3.5.0.0 64-bit installed and used for preparing hdf file.
2. Used the new 16gb hdf partitioned and formatted using Amiga OS 3.9 and it's HDToolBox with default options and has filesystem PFS3-AIO v2.3 installed, dostype PDS\3 used instead of PFS\3 to attempt direct scsi. WinUAE configured to use UAE controller and HDToolBox configured to use uaehf.device.
3. Ran HstWB Installer to install Workbench 3.1, Kickstart roms, BetterWB 4.0.3, HstWB 1.1.2, EAB WHDLoad Games AGA Menu v3.0.0 and EAB WHDLoad Packs v3.0 Games, Games_AGA and all done using UAE controller. It did this using Workbench 3.1 system files.

Running pfsdoctor experimental on dh1 device gives following output at the end of the logfile:

Dungeon 0...
Devs 0...
Colossal 0...
JimmysFantasticJourney 0...
data 0...
JimmyWhitesSnooker 0...
data 0...
Saves 0...
HD-Games 0...
ss stage 0
newpassneeded 0
valid
valid2
validating reserved 14...
validating main 3370...
validating anodes 8633...
exit
checkdirtree 0
stage 4
finishing up 2... 

pfsdoctor_exp_check_without_emeraldminescd.log

Pfsdoctor 1.6 reports the same, no errors:

So it seems to work now without errors. EmeraldMinesCD must contain something that makes PFS3 filesystem go corrupt.

[tomarkus]

I have tried to corrupt the HDF you've provided in previous comments, but I wasn't able to - all the 'fishy' games like EM, FlipFlop_v1.0.zip, HoleInOne_DataDisks_v1.1.lha and HoleInOne_v1.1.lha were successfully extracted/deleted multiple times.

Since you have probably have a nice pipeline for producing desired HDF instantly, would you be able to provide a link to 2 HDF's: the one from comment above which is not producing any errors and a HDF which has exactly 5000MB on DH1 partition? I will test them our thoroughly tomorrow. Thanks in advance!

[tomarkus]

After consideration, and going through multiple threads like this:

http://eab.abime.net/showthread.php?t=77713

It seems that original HDToolBox from WB3.9 is the best and safest option when dealing with disks, and not HDInstTools as I said before (I'm also on a learning path).

[henrikstengaard]

All right, then I will keep using Amiga OS 3.9 HDToolBox. It took some time as a had to calculate cylinders to get the exact sizes.

I have created and attached a new hdf file with dh0 at 500mb and dh1 at 5000mb:
8gb_hdf_rdb_dh0-500mb-pds3_dh1-5000mb-pds3.zip

It also includes a hddreport.txt output showing that dh1 is exactly 5000mb. From the mail thread with Toni it seems you might not need it. Anyway, here it is.

[henrikstengaard]

To partly solve this issue, I will recreate the images larger than 4GB and use dostype identifier PDS\3 instead of PFS\3. This should fix most of the corruption issues, when installing large content like EAB WHDLoad Packs Demos and Games. I know that you are currently trying different versions of PFS3 to reproduce the issue with large 4GB partitions to discover if there still are bugs with new updated versions of PFS3-AIO and lastest commercial PFS3. If so, then I will need to recreate the images once a fix would be released maybe with help from Toni Wilen or Michael Pelt.

[tomarkus]

I have managed to reproduce this issue on all versions of PFS - commercial, free, AIO, DS and non-DS. I think it would be good to just warn the users that in some edge cases >4GB partition might get corrupted at some point. Since we have 100% reproducible scenario, I will verify the bug-fix as soon as I receive a fix from Toni or Michael. Thanks for help arranging the HDF partitions.

[henrikstengaard]

Thanks for the update. It's a good idea about a warning to let users know they may experience corruption issues with >4GB images and installing unpack on amiga games like EmeraldMinesCD.lha.

I will close this issue once I have added warning messages, then we can reopen the issue later when a PFS3 bugfix would appear.

[henrikstengaard]

Warnings has now been added, if images uses harddrives larger than 4GB and are shown both when creating a new image fromtemplate and when using install modes: run, install and build self install.