Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for AppleSingle #20

Closed
oliverschmidt opened this issue Feb 19, 2018 · 19 comments
Closed

Support for AppleSingle #20

oliverschmidt opened this issue Feb 19, 2018 · 19 comments
Assignees

Comments

@oliverschmidt
Copy link

@oliverschmidt oliverschmidt commented Feb 19, 2018

Hi,

Right now there's a very nice momentum to improve the interoperability of Apple II cross development tools. The primary aspect is to have a common way to express ProDOS 8 meta data on the path from the development tool (running on a foreign file system) to the Apple II disk image management tool that writes ProDOS 8 meta data into the disk image.

cc65 and AppleCommander team up here since many years in a great way. cc65 generates a 4-byte DOS 3.3 header for BIN executable and AppleCommander has the -cc65 option to consume the 4-byte header.

However, from a conceptual POV the 4-byte DOS 3.3 header is far from optimal, especially as it doesn't contain the ProDOS 8 File Type. Fortunately Apple created already back in 1990 the AppleSingle/AppleDouble file format that can be used well as replacement for the 4-byte header.

I want drop the support for the 4-byte DOS 3.3 header in cc65 completely and switch to generation of an AppleSingle file. Of course the success of this plan heavily depends on AppleCommander to support AppleSingle files! Therefore I need to ask you to add support for AppleSingle asap.

I see several options for AppleCommander:

AppleSingle can only be supported for put operations - like it is currently with the -p variant -cc65 - or it can be supported for both put and get operations.

As AppleSingle isn't specific to cc65 and is in fact supposed to be soon supported by Merlin32 too. So from my POV its use shouldn't have a name like -cc65 that relates to cc65.

From my perspective -cc65 can be dropped from AppleCommander. However, I can imagine very well you're interested to keep to for backward compatibility reasons.

Now for the technical details:

AppleSingle is described in http://kaiser-edv.de/documents/AppleSingle_AppleDouble.pdf. Additionally there's an RFC in https://tools.ietf.org/html/rfc1740.

AppleSingle is a container for several optional "Entries" of different types. Each entry type has a specific "Entry ID". The files created by cc65 always contain exactly two entries with the Entry IDs 1 (Data Fork) and 11 (ProDOS File Info).

The actual location of entries in the file is described by a file header. This in general rather complex layout becomes pretty simple when the Entry ID 11 (the ProDOS File Info) is placed first and the Entry ID 1 (Data Fork) is placed last because the Entry ID 11 has a fixed, small size.

So from the AppleCommander perspective I see two viable approaches to handle AppleSingle files:

  1. The "correct" approach: Follow the AppleSingle spec to indentify (or even generate) all relevant Entry IDs.

  2. The "quick" approach: Presume that AppleSingle files have the layout that cc65 creates and treat them as 1:1 replacement of the 4-byte DOS 3.3 header.

For the second approach there's no need to look at any AppleSingle spec. Rather all that needs to be known is:

  • There's a header with a size of $3A bytes. After that header the rest of the file is the actual file data (just like with the old header of 4 bytes).

  • At the file offset $24/$25 there's the length of actual file data (just like with the old header at offset $02/$03). However, that value is big endian !!! If you ignored that value so far you can continue to do so.

  • At the file offset $35 there's the ProDOS File Type. It can be either $06 for a BIN file or $FF for a SYS file.

  • At the file offset $38/$39 there's the Auxiliary Type for BIN files (just like with the old header at offset $00/$01). However, that value is big endian !!!

  • At the file offset $33 there's the ProDOS Access Byte. It's set to %11000011 meaning to enable Access, Destroy, Rename, Write and Read. For me personally it's purely nice-to-have that this value is actually used. You can as well ignore it from my POV.

Again, I'd really appreciate if you could add support for AppleSingle quickly as this would help to keep up the momentum we currently have in harmonizing the files created / consumed by different Apple II tools. In order to ease that I attached an AppleSingle file created with cc65. It's a BIN file with Auxtype $803. It's an executable printing "Hello, World": https://github.com/AppleCommander/AppleCommander/files/1737098/hello.zip

Thanks in advance, Oliver

@a2geek

This comment has been minimized.

Copy link
Contributor

@a2geek a2geek commented Feb 25, 2018

I like! As we're talking about the cc65 suite of tools, am I right in assuming that the command-line ac tool is the primary target for AppleCommander? I ask because one of the fundamental problems with AppleCommander is the assumption of a physical disk (aka, some form of disk geometry). I'll have to review how we shoehorned Shrinkit files into the GUI tooling (pretty certain it dynamically allocates a disk on the fly), so maybe we can fake it in for that aspect.

@a2geek

This comment has been minimized.

Copy link
Contributor

@a2geek a2geek commented Feb 26, 2018

I remembered seeing an AppleSingle ProDOS File Type tech note at some point. Just in case it's useful: http://www.apple2online.com/web_documents/ft_e0.0001_applesingle.pdf

@a2geek

This comment has been minimized.

Copy link
Contributor

@a2geek a2geek commented Feb 26, 2018

One note regarding file name... I don't see the "Real Name" entry. I assume this is because cc65 is a compiler/assembler, so it really doesn't have one. I suspect that means AppleCommander should (a) try to use entry ID 3 if present or (b) just use the filename of the AppleSingle file. Sound like a valid approach? Thanks!

@oliverschmidt

This comment has been minimized.

Copy link
Author

@oliverschmidt oliverschmidt commented Feb 26, 2018

I like!

:-)

am I right in assuming that the command-line ac tool is the primary target for AppleCommander?

Absoulutely! This "issue" is only about the command-line ac tool.

AppleSingle ProDOS File Type tech note

That is AppleSingle version 1. We're discussing AppleSingle version 2.

I don't see the "Real Name" entry.

Correct. In general an AppleSingle file is "supposed to" have entries with the IDs 3 and 8. But after some discussion we agreed to not include those in the AppleSingle files we want to use for transporting meta data from cross dev tools to disk image tools. And after all the RFC says Each entry is optional and may or may not appear in the file. so we're conceptually on the safe side.

I assume this is because cc65 is a compiler/assembler, so it really doesn't have one.

Yep.

I suspect that means AppleCommander should (a) try to use entry ID 3 if present or (b) just use the filename of the AppleSingle file. Sound like a valid approach?

Absolutely! That's from my perspective not only a valid approach but the optimal approach.

@a2geek a2geek self-assigned this Mar 2, 2018
@a2geek

This comment has been minimized.

Copy link
Contributor

@a2geek a2geek commented Mar 2, 2018

Working through this now, a couple of notes (comment if necessary!):

  • Assuming ProDOS only. I expect to error if the disk image is DOS 3.3 and the AppleSingle file specifies ProDOS. (Especially in the cc65 case! We can translate the meta-data but the binary is likely to assume one OS over the other.) If a DOS 3.3 example shows up, then we can add support for it.
  • Ignoring dates.
  • I expect the access bits to be ignored and $C3 to be assumed.

I don't think any of this is an issue, but thought I should make a note of it.

@a2geek

This comment has been minimized.

Copy link
Contributor

@a2geek a2geek commented Mar 2, 2018

Ha! Went to look for the DOS 3.3 entry. It's MS-DOS. So apply a filter to that comment above. ;-)

@oliverschmidt

This comment has been minimized.

Copy link
Author

@oliverschmidt oliverschmidt commented Mar 2, 2018

Working through this now

:-)

the binary is likely to assume one OS over the other.

No. As long as the user/programmer keeps clear from certain features, the same binary works on both ProDOS and DOS3.3 (see http://cc65.github.io/doc/apple2.html#ss8.1). So the linker doesn't make a difference between ProDOS and DOS3.3 binaries. So placing an AppleSingle file onto a DOS3.3 image is a viable use case from cc65 perspective. E.g. the "Hello, World" I supplied works on DOS3.3 nicely.

Ignoring dates.

I guess so far you used the date of the file in the "host" file system (NTFS, ext3, ...) for the date of the file in the ProDOS file system. I'd just stick to that. The AppleSingle files created with cc65 will never have a date entry.

I expect the access bits to be ignored and $C3 to be assumed.

Using the access bits form the AppleSingle file as-is for ProDOS would seem more straight-forward from the user's perspective, but if it's simple/better to ignore it it's perfectly fine for me as the AppleSingle files created with cc65 will never ever have something diffrent than $C3.

@a2geek

This comment has been minimized.

Copy link
Contributor

@a2geek a2geek commented Mar 2, 2018

I'm not spotting a "translation" in the current code base and I don't see an Apple DOS 3.3 header in the AppleSingle. Can I assume that the DOS and ProDOS file sets up the AppleSingle header? I'll go ahead and jimmy something together to do some translation (probably TXT=>T, BAS=>A, and B for everything else).

For dates, that's more challenging. I'm following the rest of the command-line interface, so the actual binary input is piped into ac. (Meaning that I don't have those dates available...) There is, however, an AppleSingle entry ID 8 with the file creation and modification date. If cc65 were to populate that entry, ac could just use the dates. Otherwise it's just the time the ac command is executed.

I'll peak at the access bits. I didn't see anything obvious in the current ac code, so ... no promises!

@oliverschmidt

This comment has been minimized.

Copy link
Author

@oliverschmidt oliverschmidt commented Mar 2, 2018

I'm not spotting a "translation" in the current code base and I don't see an Apple DOS 3.3 header in the AppleSingle. Can I assume that the DOS and ProDOS file sets up the AppleSingle header? I'll go ahead and jimmy something together to do some translation (probably TXT=>T, BAS=>A, and B for everything else).

I'm afraid we have some misunderstanding here. Maybe the problem is that I have no overview over the AppleCommander functionality. I can only talk about the functionality relevant from the cc65-point-of-view. And from that POV the current situation is:

  • A binary file can be put both onto DOS 3.3 and ProDOS disks. If the file has no header then using -p B[IN] and providing the start address. If the file has a four-byte header then using -cc65. Either with or without header the same file works both for DOS 3.3 and ProDOS.
  • A ProDOS system can be put onto ProDOS disks using -p SYS. It has no header.

The desired situation from the cc65-POV is:

  • A binary file can be put both onto DOS 3.3 and ProDOS disks. If the file has no header then using -p B[IN] and providing the start address. If the file is an AppleSingle file then using -p or some new option without file type specification. Either AppleSingle file or not the same file works both for DOS 3.3 and ProDOS.

  • A ProDOS system can be put onto ProDOS disks. If the file has no header then using -p SYS. If the file is an AppleSingle file then using -p or some new option without file type specification.

That means that from the cc65-POV it's desirable that AppleCommander can put an AppleSingle file with an entry of ID 11 and file type $0006 onto a DOS 3.3 disk by extracting the start address form the aux type and building a DOS 3.3 four byte header from it.

@oliverschmidt

This comment has been minimized.

Copy link
Author

@oliverschmidt oliverschmidt commented Mar 2, 2018

For dates, that's more challenging.

Hm, I must admit that I can't follow you why...

Otherwise it's just the time the ac command is executed.

If that's is how AppleCommander works today then just keep to that. I don't see any reason to change something about that at all.

I didn't see anything obvious in the current ac code, so ... no promises!

If it remotely looks like becoming complicated then just keep to how AppleCommander works today

@a2geek

This comment has been minimized.

Copy link
Contributor

@a2geek a2geek commented Mar 2, 2018

Actually, no. -p is to put a file exactly as it is. No interpretation. From stdin (so piped or redirected). <filename> here is the name it will be in the disk image, not the source of the data.

-p  <imagename> <filename> <type> [[$|0x]<addr>] put stdin
    in filename on image, using file type and address [0x2000].

I'm currently modeling the -as (-applesingle?) after the old -cc65 tag. Which I should deprecate and maybe rename -dos or something. Current help text:

-cc65 <imagename> <filename> <type> put stdin with cc65 header
      in filename on image, using file type and address from header.
-as <imagename> [<filename>] put stdin with AppleSingle format
      in filename on image, using file type, address, and (optionally) name
      from the AppleSingle file.

<filename> is optional in the AppleSingle case because it may be within the AppleSingle file itself. If there is no name entry, <filename> becomes required.

Generally, I don't want to totally change the way ac is working (via stdin), so I'm following the existing pattern.

@a2geek

This comment has been minimized.

Copy link
Contributor

@a2geek a2geek commented Mar 2, 2018

Some sample runs...

  1. Create specific disk type.
  2. Import form AppleSingle sample file.
  3. List files.

Placing HELLO into a DOS 3.3 disk:

$ java -jar build/libs/AppleCommander-ac-1.4.0-BETA.jar -dos140 dos.dsk
$ cat src/test/resources/hello.applesingle.bin | java -jar build/libs/AppleCommander-ac-1.4.0-BETA.jar -as dos.dsk HELLO
$ java -jar build/libs/AppleCommander-ac-1.4.0-BETA.jar -l dos.dsk 
dos.dsk DISK VOLUME #254
  B 013 HELLO 
DOS 3.3 format; 131,840 bytes free; 11,520 bytes used.

Placing HELLO onto a ProDOS disk:

$ java -jar build/libs/AppleCommander-ac-1.4.0-BETA.jar -pro140 prodos.po WORK
$ cat src/test/resources/hello.applesingle.bin | java -jar build/libs/AppleCommander-ac-1.4.0-BETA.jar -as prodos.po HELLO
$ java -jar build/libs/AppleCommander-ac-1.4.0-BETA.jar -l prodos.po 
prodos.po /WORK/
  HELLO BIN 007 03/02/2018 03/02/2018 2,912 A=$0803 
ProDOS format; 136,192 bytes free; 7,168 bytes used.
@oliverschmidt

This comment has been minimized.

Copy link
Author

@oliverschmidt oliverschmidt commented Mar 3, 2018

Actually, no. -p is to put a file exactly as it is. No interpretation. From stdin (so piped or redirected).

Even re-reading what I wrote I don't see where I wrote anything contradicting this. Anyhow, I fully understand (and agree).

Generally, I don't want to totally change the way ac is working (via stdin), so I'm following the existing pattern.

I, again, fully agree. Sorry if I wrote something that appears to contradict this.

@oliverschmidt

This comment has been minimized.

Copy link
Author

@oliverschmidt oliverschmidt commented Mar 3, 2018

I'm currently modeling the -as (-applesingle?) after the old -cc65 tag. Which I should deprecate and maybe rename -dos or something. Current help text: [...]

That seems just great :-))

Some sample runs... [...]

Cool! Can you provide that AppleCommander-ac-1.4.0-BETA.jar to me? The AppleSingle file I posted here was created with some proof-of-concept cc65 code. With your BETA I'd be able to test a real cc65 AppleSingle implementation - and provide you with feedback...

@a2geek

This comment has been minimized.

Copy link
Contributor

@a2geek a2geek commented Mar 3, 2018

Just pushed the 1.4 BETA!!

a2geek added a commit that referenced this issue Mar 4, 2018
@oliverschmidt

This comment has been minimized.

Copy link
Author

@oliverschmidt oliverschmidt commented Mar 5, 2018

Just pushed the 1.4 BETA!!

Thanks! I checked it out and -as works exactly as expected :-)

So from my perspective 1.4 is ready for release. However, I noticed that I needed to update to Java SE 8 in order to run AppleCommander-ac-1.4.0-BETA.jar but I guess that's expected / intended.

@oliverschmidt

This comment has been minimized.

Copy link
Author

@oliverschmidt oliverschmidt commented Mar 7, 2018

I just released cc65 2.17 which now generates AppleSingle files: https://github.com/cc65/cc65/releases/tag/V2.17

Here's the announcement - of course referring AppleCommnader:
https://sourceforge.net/p/cc65/mailman/message/36247481/

@a2geek

This comment has been minimized.

Copy link
Contributor

@a2geek a2geek commented Mar 10, 2018

Closing ticket out, as 1.4.0 was just released. See https://github.com/AppleCommander/AppleCommander/releases/tag/v1-4-0

@a2geek a2geek closed this Mar 10, 2018
@oliverschmidt

This comment has been minimized.

Copy link
Author

@oliverschmidt oliverschmidt commented Mar 10, 2018

Thanks a lot for your support :-))

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.