How difficult it is to make appdmg cross-platform? #14

Open
szwacz opened this Issue Nov 22, 2013 · 48 comments

Projects

None yet
@szwacz
Contributor
szwacz commented Nov 22, 2013

Hello again.

This lib could be intergallactically awesome if dependent only on pure-js node libraries, so it can run on every OS node is running on. How difficult it is to make it this way?
You mentioned previously building .DS_Dtore.
What are the other problems to address?

If this is relatively easy I might be interested in contributing to this effort.

@LinusU
Owner
LinusU commented Nov 22, 2013

Information regarding the .DS_Store file format:

Roadmap:

  • Find SetFile replacement

We use SetFile to enable the icon-flag. Using a pure js implementation here would be awesome just in it self since SetFile is distributed with Xcode. This is only necessary for icon-support.

  • Find hdiutil replacement

We need to be able to create a dmg file from a folder, this shouldn't be too hard. hdiutil is distributed with vanilla OS X so this is only needed for support of other platforms.

  • Find bless replacement

We use the bless command to make sure that the Finder opens the image after mounting it. I would guess that this program really just sets some bits. Should be fairly easy to implement in pure js.

  • Generate .DS_Store file

This is, AFAIT, the hardest part. Right now we use apple script to open up the finder, rearrange the icons and set the background. Then sleep for some seconds which makes Finder save the changes to the .DS_Store file.

The best way to do this would is writing the file directly from the program, but we need to figure out how the file format works and what bits and bytes we need to set in order to accomplish the following:

  • Set icon size
  • Set view to icon view
  • Hide toolbar, sidebar, etc.
  • Set the background
@LinusU
Owner
LinusU commented Nov 22, 2013

I'm going to start investigating SetFile now, since that would benefit the application at large, and report back here later.

@szwacz
Contributor
szwacz commented Nov 22, 2013

I don't know if .DS_Store is actually that scary, afterall you don't have to understand the whole spec of it, you don't want to parse it, you just want to put some values in predefined binnary :)

When I read about DMG images themselves it looks like more of a hedache http://newosxbook.com/DMG.html
Since this is fully proprietary standard it is only reverse-engineered to other platforms. There is something for linux: http://serverhorror.wordpress.com/2011/02/26/create-dmg-images-for-os-x-on-linux/ but I can find nothing for windows.

@LinusU
Owner
LinusU commented Nov 22, 2013

Ohh, I actually thought that dmg images was just a raw binary file with a file system... I have some reading up to do :)

@LinusU
Owner
LinusU commented Nov 22, 2013

The SetFile is a nightmare as well, but it seems like it's just really an extended attributes of 32 bytes (com.apple.FinderInfo).

This is the output on a clean folder with only the custom icon applied.

xattr -p com.apple.FinderInfo /tmp/test1607
00 00 00 00 00 00 00 00 04 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

This project might have some relevant info as well: https://code.google.com/p/profuse/wiki/XattrUtility

@LinusU
Owner
LinusU commented Nov 22, 2013

Fun fun fun, I think I actually have a way to do SetFile cross-platform.

> x = require('xattr')
{ list: [Function], set: [Function] }
> x.list
[Function]
> x.list.toString()
'function () { [native code] }'
> x.list('/tmp/test1607')
{ 'com.apple.FinderInfo': '\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0004\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000' }
> x.list('/tmp/test1607')['com.apple.FinderInfo']
'\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0004\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000'
> var target = x.list('/tmp/test1607')['com.apple.FinderInfo']
undefined
> target
'\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0004\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000'
> x.set('/tmp/test1734', 'com.apple.FinderInfo', target)
true
> 
  • /tmp/test1607 is a folder which I have used SetFile on
  • /tmp/test1734 is a folder I made with mkdir
@LinusU
Owner
LinusU commented Nov 22, 2013

Okay, so I don't know why this works (in that I don't know what the different bits mean) but this should create a buffer that can be written to the com.apple.FinderInfo xattr.

var buf = new Buffer(32);
buf.fill(0);
buf.writeUInt8(4, 8);
return buf.toString('binary');
@LinusU
Owner
LinusU commented Nov 22, 2013

Seems like most of the things we need for .DS_Store is documented on the mozilla wiki.

Structure fwi0 type blob is Finder window information. Known length is 0x10 (16). The data is first four two-byte values representing the top, left, bottom, and right edges of the rect defining the content area of the window. The next four bytes represent the view of the window, "icnv" is icon view. The next four bytes are unknown.

Structure fwvh type shor is Finder window vertical height. If present, it overrides the height defined by the rect in fwi0. The Finder seems to create these (at least on 10.4) even though it will do the right thing for window height with only an fwi0 around, perhaps this is because the stored height is weird when accounting for toolbars and status bars.

Structure icvt type shor is icon view text label (filename) size.

Structure icvo type blob is icon view options. Known length 0x12 (18), first 4 bytes "icvo", then 8 unknown bytes (flags?), then 2 bytes corresponding to the selected icon view size, then 4 unknown bytes (0x6e 6f 6e 65) (the text "none", guess that this is the "keep arranged by" setting?)

Structure Iloc type blob is icon location for the last-identified file. Length is 0x10 (16), two 4-byte values representing the horizontal and vertical positions of the icon's center (not top-left). (Then, 6 bytes 0xff and 2 bytes 0?) For the purposes of the center, the icon only is taken into account, not any label. The icon's size comes from the icvo blob.

Structure BKGD type blob known length 0x0c (12) is for the background. It contains a reference to another strucutre, the first four bytes are PctB, followed by four bytes indicating the length of the referenced pict structure (same as the pict's length), then 00 00 00 13 (always the same?)

Structure pict type blob, length dependent on contents. This is for the background, along with BKGD. The contents: first, two empty bytes (00 00) followed by 4 bytes giving the length of the entire structure again, same as the length following "pictblob" in the header. Then, 4 bytes, 00 02 00 00. Then, 1 byte giving the length of the volume name. Then, 31 bytes to hold the volume name, unused bytes are 00...

This seems easy! 🍻

@LinusU
Owner
LinusU commented Nov 23, 2013

I tried looking at the FinderInfo after running bless and most 00 where replaced by 20, no idea why thought 🎊

xattr -p com.apple.FinderInfo /tmp/test1734
20 20 20 20 20 20 20 20 04 20 20 20 20 20 20 20
20 20 20 20 00 00 00 00 20 20 20 20 20 20 20 20

GetFileInfo shows no difference with or without bless...

@LinusU
Owner
LinusU commented Jan 2, 2014

I have now stopped using seticonflag so this should be one step closer to success!

@al45tair

The FinderInfo data is documented (see the man page for getattrlist on a Mac OS X system), so there’s really no need to guess at its contents.

FWIW, I’ve already got portable Python code for reading/writing/editing .DS_Store files (see the ds_store module on PyPI), and semi-portable code for generating the Alias record you need to set background images (see mac_alias on PyPI). There are some issues with making the alias code fully portable (namely it needs to be able to extract HFS+ catalog node IDs from the filesystem somehow in order to generate the alias record correctly).

There’s also my dmgbuild tool, which is based on the above; that currently uses SetFile, though I have in mind that I might ditch that some time soon.

@LinusU
Owner
LinusU commented Mar 14, 2014

Cool, I'm gonna have a look when I have time and updated my code.

Had a quick look at your documentation, it seems to be well done, keep up the good work! Feel free to contact me at any time if you need any help or want to discuss some file format :)

@adam-lynch

This would be unbelievable!

@felicienfrancois

For hdutil replacement, maybe hfsutils can do the job.
http://serverhorror.wordpress.com/2011/02/26/create-dmg-images-for-os-x-on-linux/

hfsutils is licensed under GPLv2 license and seems to have ports for windows:
http://www.mars.org/home/rob/proj/hfs/

@felicienfrancois

There is also https://github.com/planetbeing/libdmg-hfsplus which supports HFS+

It seems to be buildable for both linux and windows: https://github.com/planetbeing/libdmg-hfsplus/blob/master/CMakeLists.txt

@stevenvachon

+1

@guidograzioli

You can also create a dmg (uncompressed tough) on linux, with an iso9660/udf hybrid filesystem, with the command:

# genisoimage -V Name -D -R -apple -no-pad -o Name.dmg directory
@al45tair

@guidograzioli There’s nothing special about uncompressed DMGs — they’re just raw disk image files, so you can equally create one using dd and the appropriate mkfs command (you don’t need to use genisoimage). You can even mount uncompressed DMG files (assuming you have the correct filesystem drivers in the kernel) using mount -oloop /path/to/disk/image.dmg /mount/point.

It’s also worth pointing out that the Mac is able to handle a number of filesystems, besides HFS+, out of the box, including FAT, NTFS (read-only) and ExFAT as well as a variant of UFS. There’s nothing stopping you from creating DMGs using those filesystems; modern Mac software doesn’t really use many of the special features of HFS+ anyway, and most users won’t notice and don’t care anyway, so you won’t be losing much by doing that, and no need for an HFS driver.

@LinusU LinusU referenced this issue in electron-userland/electron-packager Jun 18, 2015
Closed

provide way to create "installers" (.dmg, .msi, .deb etc) #33

@bundyo
bundyo commented Jun 30, 2015

I managed to build a working dmg under Linux, using genisoimage and the dmg tool from this repo to compress it (its image building doesn't work, but compresses fine).

Turned out since OS X and Linux are both unix based, you only need to have in the root a hidden .background folder with background.png in it for the image, the said .DS_Store with the background specified (which I built on a real OS X in order to use it), a .VolumeIcon.icns holding the icons and a normal symlink called Applications to /Applications. And of course the .app folder. Something like this:

image/ -> | .background/     -> | background.png
          | MyApp.app/       -> | ...
          | Applications     ~ /Applications
          | .DS_Store
          | .VolumeIcon.icns

Then I used genisoimage like this (where image/ is the dmg root above):

genisoimage -D -V "App Name" -no-pad -r -apple -file-mode 0755 -o ./my_temp.dmg image/

And dmg like this:

dmg dmg ./my_temp.dmg ./my_app.dmg

However afterwards I've hit another roadblock - codesign, so I moved my build to an OS X machine. :(

I hope this helps.

@LinusU
Owner
LinusU commented Jun 30, 2015

@bundyo The .DS_Store can be generated even on linux with node-ds-store. The link to /Application is usually an alias instead of a symlink, but I guess that maybe it doesn't really matter...

@al45tair

@LinusU It’s better to use a symlink rather than an alias for /Applications IMO. That’s what we’ve always done.

@bundyo
bundyo commented Jun 30, 2015

Thanks. @LinusU I basically did the same the first time, though with GHex :)

@bundyo
bundyo commented Jun 30, 2015

Is there an alternative to codesign, aside from ldid, which AFAIK doesn't work on Linux or for OS X apps?

@AlicanC
AlicanC commented Oct 7, 2015

Dropping util.sh() should also be in the checklist.

You can use fs-extra's copy() for util.sh('cp', ['-R', resolvePath(entry.path), finalPath], cb).

@kangas kangas referenced this issue in mongodb-js/electron-installer-dmg Nov 5, 2015
Closed

Error: Must be run on OSX #1

@cumajkeee

Hi, @LinusU, according status you found how to hide sidebar, toolbar in installer window, can you please somehow describe how to do it? Thanks in advance.

@LinusU
Owner
LinusU commented Dec 3, 2015

@cumajkeee This bug is for tracking cross-platform support. I've never seen the sidebar or toolbar show up when using appdmg.

@havenchyk

Hi @LinusU! I have the same problem as @cumajkeee.

Could you please suggest the way to hide unnecessary parts like sidebar or toolbar?

Yes, I know that background picture is not exists, but with the image we have the same result.

wrong finder

@LinusU
Owner
LinusU commented Dec 3, 2015

Please open a new issue and I'll take a look at it, this is an issue about cross-platform support

@havenchyk

@LinusU sure, sorry for this. Opened #79

@LinusU
Owner
LinusU commented Dec 3, 2015

No problem, thank you. I'll have a look this weekend

@KevinMartin

@LinusU , what is missing for cross platform support?

@LinusU
Owner
LinusU commented Dec 3, 2015
  • A way to make an .dmg file on Linux/Windows
  • macos-alias for Linux/Windows (should be easy)
  • bless replacement (might not be that important)

I think that should be it, there was some work by someone working on packaging electron apps. I think that they had solved it.

@stevenvachon

Get it done, yo ;)

@LinusU
Owner
LinusU commented Dec 3, 2015

Haha, pull requests welcome ;)

@medfreeman

I wonder if dmg generation could work with exFat (works on all platforms RW), using dd and mkfs on linux, and dd,FileDisk on windows (both are under GPL) with DiskPart. I'll try this mid-january.

@develar develar referenced this issue in electron-userland/electron-builder Feb 12, 2016
Closed

Can't build OS X bundle on linux #143

@thanpolas

Is there a way to not throw an error on npm install so as to be able to build our apps on CI environments which typically use ubuntu?

No appdmg method will be called on those environments.

What do you do today to test your apps that require appdmg?

@jakubzitny

@thanpolas add it to optionalDependencies

"optionalDependencies": {
  "appdmg": "^0.3.5"
}
@johnhaley81

@jakubzitny doesn't work if you shrinkwrap :( npm/npm#2679

@BenjaminVanRyseghem

@thanpolas maybe a bit late, but what I did was to put appdmg as an optional dependency.

it now looks like

  "devDependencies": {
    // ...
  },
  "optionalDependencies": {
    "appdmg": "^0.3.5"
  },
  "dependencies": {
    // ...
  }
@lyssdod
lyssdod commented Sep 3, 2016 edited

@LinusU What's the current status of this? Is electron-userland/electron-builder#14 did help in any way?

@develar
Contributor
develar commented Sep 3, 2016

@lyssdod You can sign app only on macOS. And unsigned app is blocked. So, for what you need to build DMG?

@lyssdod
lyssdod commented Sep 3, 2016

@develar I'm trying to setup a multiplatform build via Jenkins using electron-builder. I also have a spare mac which can be used for signing. Would it be simpler to build mac version on mac?

@develar
Contributor
develar commented Sep 3, 2016

@lyssdod Please see https://github.com/electron-userland/electron-builder/wiki/Multi-Platform-Build

I also have a spare mac which can be used for signing. Would it be simpler to build mac version on mac?

Will be simpler and more robust :) Yes, for example, in my company we do use separate servers to sign products, but only due to security reasons. So, I suggest just use mac to build version for mac. Or use Travis/CircleCI.

@lyssdod
lyssdod commented Sep 3, 2016

@develar Thanks!

@develar develar referenced this issue in electron-userland/electron-builder Oct 10, 2016
Closed

Mac build failing with spawn hdiutil ENOENT #811

@xblox
xblox commented Oct 10, 2016

on linux you have :
git clone https://github.com/hamstergene/libdmg-hfsplus
cd libdmg-hfsplus && cmake . && make && cd dmg
./dmg --help

./dmg dmg project-$(VERSION)-uncompressed.dmg project-$(VERSION).dmg

just tried, works funtastic

@develar
Contributor
develar commented Oct 10, 2016 edited

@gbaumgart ...and then your app will be blocked in any case because unsigned ;) If you can use unsigned app (i.e. it is ok for your users) — well, just use zip or tar.gz And even more — Apple recommends to sign even DMG since macOS Sierra (electron-builder doesn't sign DMG files yet, but definitely it will be fixed this autumn).

@xblox
xblox commented Oct 10, 2016

@develar, yeah, coding for osx always sucked

@quorten
quorten commented Nov 15, 2016

By the way, we have the source code for bless right here:

https://opensource.apple.com/source/bless/bless-105/

Thus, if there are any doubts about the functionality of bless, we can look through the source code to verify the behavior. Indeed, from looking at the source myself, some parts of the code merely set/clear words to perform the desired operations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment