files created after 20120731 seem to fail acc. wiki

[roelandjansen]

according to https://en.wikipedia.org/wiki/PC-MOS/386

"t appears there is a year 2000 like issue in this product, only happening on 31 July 2012 rather than the year 2000. Files created on the system after this time will no longer work."

This may need some testing and checks. Maybe issue #1 is also related.

[andrewbird]

Installation from 1_MOS5userSystemSN0530041410.zip fails if the system date is current. It seems that hdsetup can't read a file it created earlier. Deleting the file it complains about, setting the clock back (I used 1993), and restarting install allows installation to proceed.

[the-grue]

I have been looking into this particular bug and have made some interesting (though not necessarily related to the problem) observations:

1. 2099 - 1980 = 119 years
2. 1/1/1980 - 7/31/2012 = 11900 days or 1700 weeks
3. 2012 - 1980 = 32 years (rollover to bit position 6)
4. August = 8th month (rollover to bit position 4)

Item 1 is standard DOS, so I don't see that as particularly problematic, though the coincidence of item 2 being 100 x item 1 is interesting. I just put in the 1700 weeks because the date converter offered that up along with 285600 hours.

I don't think 3 alone causes any problems, or it would have started having issues on 1/1/2012. Item 4 may or may not be of interest, but I am not ruling it out.

Of course, I expect it is something different completely, or I would have come across the smoking gun already.

I wrote some simple MOS programs and using a FAT reader I wrote years ago on Linux I can see that starting on 8/1/2012 writes to the directories seems to mess up the directory entries and FAT cluster pointers. You can see the files in a directory listing, you can delete the file, but if you try to open the file it returns "file not found".

@andrewbird to get around the date issue in the installation, I just added "date 08-24-1995" as the first line of autoexec.bat. (Date chosen as a bit of a joke.)

[andrewbird]

@the-grue I see you are working on the date problem. I fixed a (maybe) similar thing in dosemu's fat image creation dosemu2/dosemu2#759 where the dates after 2012 rolled over to 1980 because the 7 bit year field was assumed to be 5 bits wide. I started looking at PC-MOS for something similar, but I'm having trouble creating a bootable $$mos.sys, have you managed to boot a PC or Dosemu with a newly built $$mos.sys?

[ghost]

The PCMOS386-9-user.img in IMAGES boots from floppy. Then I built the kernel $$MOS.SYS from source, copied it to floppy, and now it's only the 60 minute demo version. Still boots though.

I floppy booted into MOS with a 1980 date, created a short file "echo abc>z.txt", booted back into DOS 6.22, read the file, all good. Repeated the procedure with a 2019 date, DOS 6..22 can read it, but the contents are garbage. SCANDISK does not complain though, so there is no FAT corruption that SCANDISK understands. There may be some, but SCANDISK knows nothing about it.

I'll have to try looking at the raw bytes with a disk editor.

[andrewbird]

@src153 Thanks for the confirmation that build works. I realised I'd forgotten to fix the timestamp on it before testing.
Incidentally the problem is in the readpath too. I copied a file on disk outside of MOS, touched it to get the current date, and it's unreadable by MOS.

[C:\]dir old*.*
 
FILENAME.EXT       SIZE  ----UPDATED---- ATTRB CLS USER  ----CREATED----
 
OLDCMD  .BAT        180   4-07-95   9:13
OLDCMD  .NEW        180   1-16-19  19:19
 
DISKID  DXXXXS MOS          360 BYTES     2 FILES     8,843,264 BYTES REMAINING
 
[C:\]type oldcmd.bat
COPY C:\PCMOS\$$SHELL.OLD C:\$$SHELL.SYS
COPY C:\PCMOS\COMMAND.OLD C:\COMMAND.COM
COPY C:\PCMOS\$$SHELL.OLD C:\PCMOS\$$SHELL.SYS
COPY C:\PCMOS\COMMAND.OLD C:\PCMOS\COMMAND.COM
 
[C:\]type oldcmd.new
Cannot find file

[the-grue]

Yeah, I have been down that path already and I am currently looking into the int 38h/int d4h and how MOS stores the user and class information on disk. Based on what I see of the data structures, they may have used a formerly reserved section of the directory entry to store this information (10 bytes between attribute and update time). Later versions of DOS use this area for creation time/date, last access date, etc.

I believe the file unreadability may have something to do with the class entry and the security features, but I haven't gotten a good enough understanding yet of how all that works. Or why it would suddenly start on 8/1/2012.

You'll notice the problem when you see garbage under the CLS and USER fields in a directory listing under MOS for files created after 8/1/2012 OR files created under newer DOS/Windows/Linux/etc.

As for the 5 bit vs 7 bit year, I thought of this and put it aside figuring it would have manifested on 1/1/2012. Unless it is a combination of year and month somehow clobbering something.

Which gives me an idea!!!

Back in a bit.

[the-grue]

Well, my hunch didn't pan out. Looks like MSDOS formatted filesystems and MOS formatted filesystems don't behave any different.

Here's an example of a dump of two different files and their contents. TEST.TXT is dated 08/01/2012 and TEST2.TXT is dated 01/17/2002. Both have the same contents, notice the first block for TEST.TXT is garbage and the second one is fine.

0023000 T E S T T X T \0 \0 \0 \0
0023020 \0 \0 227 b 1 N 227 b 1 N 002 \0 023 \0 \0 \0
0023040 T E S T 2 T X T \0 \0 \0 \0
0023060 \0 \0 274 b 1 , 274 b 1 , 003 \0 023 \0 \0 \0
0023100 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
*
0041000 020 303 310 312 345 023 315 350 026 330 030 355 337 356 360 037
0041020 036 374 372 366 366 366 366 366 366 366 366 366 366 366 366 366
0041040 366 366 366 366 366 366 366 366 366 366 366 366 366 366 366 366
*
0042000 " T h i s i s a t e s t "
0042020 \r \n 366 366 366 366 366 366 366 366 366 366 366 366 366
0042040 366 366 366 366 366 366 366 366 366 366 366 366 366 366 366 366

[ghost]

Working backwards from the error message, I've folllowed the trail to mosfun3d in mosfun39.asm. I see it makes a call to hl_open. I've not looked at that yet, but I'm guessing there's no obvious problem, or you would have already found it.

Single step debugging starting with the call to hl_open would be nice. I wonder if there is a way to serial port debug it with vmware. What have you tried? Any suggestions?

[roelandjansen]

to be honest, I never ever tried that nor thought about it.

It seems that workstation can do at least with version 5.5 and now that we're at 15.xx I blindly assume it still works this way.

I only googled, not read much since I'm at a short break before I continue working:

https://www.vmware.com/support/ws55/doc/ws_devices_serial_advanced_example_debugging.html
https://briolidz.wordpress.com/2012/03/28/windows-driver-debugging-with-windbg-and-vmware/

[roelandjansen]

good work so far by the way!

[ghost]

VMWARE debugging sounds like it uses the tools included with the guest, like the Windows kernel debugger or GDB for linux. DOS has no such debugger that I know of. Does PC-MOS? In one of the ASM source files I saw some mention of debugging, but not sure what that's about. Maybe you know more. And there is Softice for DOS but I don't know if it works with a kernel or not.

The PC-MOS kernel makefile builds each ASM like so:

public $<
masm $*
del $*.pub

I suspect public was a periscope debugger utility. But I don't know what the purpose was, because removing it makes no difference in the object produced by masm.

Meanwhile, I found the Bochs project. It has its own internal debugger, with a GUI interface on a Windows host. I think I will try that first.

[roelandjansen]

if mem serves right, it indeed was a hw periscope debugger.

[ghost]

Before learning Bochs debugger, which is a sub project in itself, I need to read more code to have a better idea where to focus.

hl_open is not very big, I don't suspect the problem is there. But it calls pathfind in mosfutil.asm, which is not so simple. There is also pathcomp, findnm, and nextnm. A lot of code where things could go wrong.

If you could just compile it like a user app and throw it in Turbo Debugger, it would be much easier. With the Bochs debugger, you have no symbolic names for variables. You have to step line by line and look at the reverse assembled instructions to match them up with where you are in the source code.

The plan of attack is to step through the code with an old date where it succeeds, and then repeat with a current date where it fails. Where they diverge, that's where things go wrong, and then you try to understand why, by looking at registers and variables.

This could take some time.

[ghost]

PC-MOS does have a DEBUG command. I suppose I could write a short ASM program which opens a file, and start it from DEBUG. I guess it would trace into the kernel. That would be as good as using Bochs, since either way, there are no symbolic variable names.

Anyone with better ideas, please enlighten.

[the-grue]

debug has worked for me so far. It has been a slow manual process, as you have seen. Single step, compare where the jumps go, what variables are being referred to, etc in the source files and move on. I am refreshing my memory on how td works so I can use that or cv to do debugging within MOS. It would be cool if we could somehow tie the code files used in the kernel to an active debug session within the OS, but I am not aware how to do that.

My guess is that this is how the Periscope debugger was used. Some of the tools in /bin have periscope banners, such as public and ts. From what I have read of periscope, it was an actual card you put in the computer to do debugging and there were associated code tools, but that's about all I have found on that subject.

[sub205]

Strange behaviour, even a disk-image written by windows with 1999 timestamps refuses to work.
Any way to get around this?

I bootet from the img, wrote the 2 disks to images (not bootable) after forcing timestamps to 1999, but even this is not working for me.

[ghost]

I don't understand. Can you provide more context, and details?

[sub205]

Bootet PCem with PCMOS386-9-user.img as DriveA image.
Created a floppy image with the files of 1_MOS5userSystemSN0530041410.zip.
Did not work as, because files on diskimage got current date (modified).
Changed all timestamps to 01/24/1999 and rewrote image, still same problem.
Systemdate was 01/01/1999.

What is the correct way of creating the images/floppies to install PCMOS with the 2012-bug?

[sub205]

tried something else, still fails.

mkfs.msdos -C PCMOS1.img 1440
mount PCMOS1.img g -o loop

unzipped 1_MOS5usersystem.zip into it
unmount
loaded image in PCem

"install" just dropped me back to the prompt
"hdsetup" hangs forever

EDIT: the strange "user" makes me think that PCMOS might some things that break FAT-compatibility.

[ghost]

I did not use linux or DOSEMU for this, so I can't analyze what you posted. What I did was:

I got the PCMOS386-9-user.img onto a DOS 6.22 real hardware computer, and copied that image onto floppy. DOS 6.22 does not have a diskcopy command, but Norton Commander can do it, so I use that.

Linux DD should also work, for copying the image to floppy.

Then I deleted all the files from the floppy, leaving myself with a bootable floppy. Then I copied the system disk files from SHIPMOS onto the floppy. Then I used a 2nd floppy and copied auxfiles.exe to it. DO NOT expand that file, PC-MOS install does it automatically during the install.

I think there is some official way to create a bootable system floppy, but I did not spend time learning it, so I improvised as above.

As for booting on PCem, I don't know enough about it to help. There is also the Bochs project if you want to try that.

[ghost]

Huh, I just looked in my DOS 6.22 directory and there is a DISKCOPY command. I never knew.

Hmmm ... looks like it only does floppy to floppy copy though, no image files. That makes it less useful.

[the-grue]

Yep, it sure does. Google VFAT or FAT32 and you will see how MS used some of the 10 reserved bytes in the directory entry that clobber stuff TSL took for their user/class playground. Also, some entries may be incompatible with later versions of DOS. Then there's the class byte, which I believe does something with on-disk encryption, but haven't zeroed in on that yet. Lots of stuff I am sifting through to try to find the root of the 2012 bug.

This is one reason why I am focusing on my DOS 5 based build environments at the moment instead of using 6.22 or Windows 3.1 or Windows 95/98.

[ghost]

I don't think we should supprt multi boot environments where a DOS partition is exposed to VFAT or FAT32 mounts. Ray Duncan's Advanced MS-DOS says on page 184, those bytes are reserved. That should be the final word on the matter. If your DOS FAT directories get polluted by a VFAT mount, I coded a directory defrag program that prunes all that garbage and fixes things back to DOS 8.3 names only. I will be glad to share the source freely.

[ghost]

BTW, it's well tested, I use it often. It only supports FAT16, not FAT12. The regular DOS defrag can handle FAT12. I wrote it because my FAT16 partition is so big, DOS defrag runs out of memory and quits. Mine does not defrag files, only directories. It also sorts the names to put them in alphabetical order.

[the-grue]

I will be glad to share the source freely.

I'll take a copy. My e-mail is in the git log.

[the-grue]

On the topic of classes and encryption:

MOS-D191 Page 9-19 "Security"
Precautions for Security Users

When you invoke security, every file having a non-blank class is stored in an encrypted state.

This manual is already paying for itself.

[ghost]

I will be glad to share the source freely.
I'll take a copy. My e-mail is in the git log.

I looked at commits but did not see it. I've not cloned a repo, so maybe it's not accessible to me. But it's not a big file, I will post it for all:

df.zip

[ghost]

DOS directories need defragging because:

a) Deleted directory entries leave empty slots in the list

b) Directories can span multiple clusters, just like files. When a directory contains more than 1,024 entries (may vary, but 1,024 is typical of FAT16), it spans multiple clusters. If you delete many files, the directory becomes inefficient. The multiple clusters should be compacted.

The code that does the compacting updates the FAT cluster chain and prunes the list. Can this be corrupted if killed in progress? I don't remember! I would have to audit the code again, to know for sure.

It's unusual for a directory to have that many entries, so the danger is minimal, but I mention it to forewarn. In my own setup, I don't worry about it, because I can quickly recreate my entire DOS partition by booting into linux and using rsync to copy the files from my server. If you don't have a similar recovery capability, don't rely on my code, I only wrote it for my use, and the documentation is very sparse.

[the-grue]

The clobbering actually turned out to be encryption based on a class entry. It was an observation before I knew about the class security and encryption.

[ghost]

In mosfutl4.asm, below label openf3, I see

mov al,[gfbclass] ; class for file
call classlvl

The success case moves 00h into al
The failure case moves 41h (ascii 'A') into al

It comes from gfbclass, so somehow, gfbclass gets a bogus value in the failure case.

That's all I know ATM, but I'm working backwards, slowly, toward the cause. I hope.

[the-grue]

I found it! Well, at least I think I did! The change I made seems to be working now! Still doing some testing...

[ghost]

So what is the fix?

[the-grue]

In mosfutl4.asm, below label openf3, I see

You are on the same track as me and a few steps behind. The bug is a little farther down in the file in the makegfb function. In summary, they read the file date into AX then clear al, copy the class into al, and do a comparison against ah! The bit shift in ah with the year in combination to the shift into ah by the high bit of the month causes it to match a class and in turn enable encryption of the files and security so you can't access the file anymore. It shouldn't be checking against ah since the class is in al, so I modified it to clear AX and copy the class into ah as it appears was intended. Here's a diff:

@@ -1275,8 +1275,8 @@ mgx1:
; systems will modified the class bits, if not 'A' - 'Z' range
; mark file with null class

•   xor     al,al                   ; default to space
  

•   mov     al,byte ptr [si+dbbbuf+dclass0]
  

•   xor     ax,ax                   ; default to space^M
  

•   mov     ah,byte ptr [si+dbbbuf+dclass0]^M
    cmp     ah,'A'
    jb      mgfbc1
    cmp     ah,'Z'
  

Care to add the change, build, and help test? I'll put up a patch for the file.

[the-grue]

Just rebuilt the software with today's date and everything is still working fine.

I pushed a patch from my fix2012bug branch that applies the changes to all copies of MOSFUTL4.ASM across the repo.

We need more testing though...

[ghost]

Before testing, I will look at the code. Moving the date into AX and the class into AL makes no sense. I want to know more about it. What's a GFB?

[the-grue]

Global file block. It is defined in MOSGFB.INC. When you look at the code, just above my change, you'll see that the date is copied from the dbbbuf into AX and then from AX into the gfbdate. The last value stored in AX prior to the class stuff is the date in directory format.

    mov     ax,word ptr [si+dbbbuf+ddate]  ; file date
    mov     [gfbdate],ax

But I wouldn't expect less, as far as wanting to look over the code before accepting it. Look it over, make sure you understand it, and let me know if you have any questions.

I had pretty much given up on the debugger and was grepping and reading code files to track it down. I used the debugger to write small programs and test all of the date/time related functions to rule them out. After that, it would only crash or hang my system when I tried to single step through the kernel.

[ghost]

I need a little time. This bug went unfixed for a year. No need to rush it. Unless the world ends tomorrow. And then it's moot.

I grep and read too, to get started. But without seeing the data and code at the same time, in a debugger, the job is harder than it needs to be. Of course with DEBUG it's still hard. That's why I looked for something better. Bochs debugger is the answer. I can start two instances at the same time, using two identical .vmdk files, and I have a magic breakpoint in the code where I want it to stop. Once there, I step each instance one line at a time, keeping them in sync, looking for a divergent code path or different data in the registers. It helps me focus on the problem quickly, like a binary search.

I don't know how far behind you I was, but it was only a matter of time. And I've only been looking at this a few days. Bochs debugger for Windows (the GUI version). Highly recommended.

[the-grue]

If you want, throw an int 3 breakpoint into the code right above the change I made and see what it does. Or what you said was the bochs breakpoint, that would probably work better.

[ghost]

Yes I will do that. I may not be smart, but I use smart tools.

[the-grue]

I don't know how far behind you I was, but it was only a matter of time. And I've only been looking at this a few days.

I am giving you props. You were in the same file where I found it, so it wouldn't have been too long unless you got swept off into another file chasing function calls, which I have done several times.

Considering we may have been 2 of maybe a handful of people who have tried to solve this puzzle...

Not bad for a few days work!

[the-grue]

I have had success rebuilding everything and am moving on to testing tools and applications. Maybe even Windows. I saw mention of Windows 3 in the source code and I have the disks floating around here somewhere...

[ghost]

No need for props. It was the tools, not me. I'm not in this for praise, friends, or any other psychological reasons. I just enjoy the code and its discovery. And as I said earlier, this project goes NOWHERE until the date bug is fixed. So that's my motivation.

Speaking of smart tools, I looked at the man page for git and it says "git - the stupid content tracker". I realize the idea of git is to track content and not containers, but if that means you lose original file dates from a DOS archive, I may run a parallel SVN just to make myself happy. And I don't know much about SVN either. I have a lot to learn.

[ghost]

The bit shift in ah with the year in combination to the shift into ah by the high bit of the month causes it to match a class and in turn enable encryption of the files and security so you can't access the file anymore

The patch is only two lines. I was looking everywhere for a "bit shift", then I realized you mean the data, which takes on "shifted" values due to the advanced date. Got it. Translating programmer code-think into English is not easy, I understand. That's why documentation is sometimes lacking.

The first time I read your post, I felt dizzy. But now that I understand, the patch sounds correct. However, I want to verify it, by playing around with the failure vs success bit patterns and walking them through the code. But that's just for my amusement, and it may take a while, so don't wait for my vote, if you're satisfied with the patch.

[the-grue]

@roelandjansen do we feel this can be closed or would you like to have a binary patch for the "as shipped" $$MOS.SYS?

[sub205]

A binary patch is always a good idea to help other people getting started with a project. The $$MOS.SYS we've got in the repo is broken, so why not fix it?

We have an image for "quick start" (PCMOS386-9-user.img and the other ZIPs) - a patch there would lower the bar for everyone.

[ghost]

The 9 user image boots, but it has the sizes of $$shell.sys vs command.com reversed, as compared to the 5 user image from SHIPMOS. It also has no auxiliary disk, which is strange. Until we understand why, it would be better to take the 5 user SHIPMOS file directories and create the 2 bootable images to help people get started.

[ghost]

@the-grue Your "Code Audit against v5.01 distribution disks"

Does "distribution disks" mean SHIPMOS? I never thought to ask before. Or maybe I did, and forgot.

[ghost]

lower the bar for everyone

Talk about a big job. The main project directory, and especially SOURCES/src, now that it has two brand new directories, is a bewildering mess. I kind of know my way around now, but somebody looking at it for the first time will not.

If I had my way, I would take the whole project and put it an ARCHIVE directory, leaving the project root empty except for the ARCHIVE directory. Then, as the finished pieces become available for actual use, put them in the root directory which is now clean and easy to navigate.

Talk about a big job. You can say that again, heh.

[sub205]

That's a good point!
First thing would be a diff of the 5 different sourcetrees, then sticking to one and rebuilding a tidy root with it.

Does anyone know right now what they are about?

[sub205]

BTW: at some point we should rename this repo or just create a new one, because the name itself implies version 5.01.

[ghost]

@the-grue has already created "mos501src" and "latest" which are basically 501 and 502, which indicates where they originated: kernel vs mos5src. However that does NOT account for all the other stuff outside the kernel, which is yet to be tested and understood.

Does that sound right to you, @the-grue?

Trying to clean all that up and lower the bar, is a long term job.

[the-grue]

Yes, that is correct. We could move jim, mossrc1, mossrc2, and mos5src to an archive directory. The rest of the code directories would need to be merged into the latest tree along with appropriate build scripts.

[the-grue]

A binary patch is always a good idea to help other people getting started with a project. The $$MOS.SYS we've got in the repo is broken, so why not fix it?

We have an image for "quick start" (PCMOS386-9-user.img and the other ZIPs) - a patch there would lower the bar for everyone.

I started with the 9 user image and will follow up with the other ones. Added branch binpatch2012 to my repo to track it, but here is the .img file for testing (forced to zip it, github rules):

PCMOS386-9-user-patched.zip

As suggested in previous posts, I won't do a PR until folks have a chance to test this.

Enjoy!

[ghost]

would you like to have a binary patch for the "as shipped" $$MOS.SYS?

You've done a lot of work on this project. But if you try to do everything, you'll burn yourself out. People who want things, can put in the effort themselves.

When I muse about how I would do such and such, I'm just thinking out loud. If I really want it done, I will break a sweat and do it. Don't let anyone take advantage of your generosity and willingness to work.

The source is the important thing. If binary images are broken, I don't care. If somebody wants them fixed, let them do the work. Save your strength, energy, and time for the real challenge: source code.

FWIW.

[ghost]

Fixed, closing