When "ditto" fails to extract an archive in PkgPayloadUnpacker, use "aa" if present

[grahampugh]

Modern Apple Archives fail to be extracted using ditto. Case in point is TeamViewer.pkg in justinrimmel-recipes.

In Slack's #autopkg channel we have established that the aa command can do the job instead. In fact, it seems to also be able to extract the Payload archives that are not modern Apple Archives but CPIO archive payloads.

However, aa only appears to have been introduced on Big Sur, so can't replace ditto just yet.

This PR makes PkgPayloadUnpacker try aa only if ditto fails and aa is found at /usr/bin/aa. If aa is not present, the ditto failure is raised as a ProcessorError.

[nmcspadden]

I would prefer we use subprocess.run() instead of Popen, but this is not a blocker.

[nmcspadden]

Please prefer subprocess.run() instead of Popen.

[grahampugh]

Do you have another example from within autokglib so I can follow the style? All I did was copy the existing code for the ditto proc command.

[nmcspadden]

@grahampugh Some other examples:

autopkg/Scripts/bulk_repo_add.py

Line 79 in 7ba6e9d

subprocess.run(cmd, check=False, capture_output=True)

autopkg/Code/autopkglib/URLGetter.py

Line 169 in 71a1737

result = subprocess.run(

[nmcspadden]

Can you show some output (doesn't have to be the whole output of a run) showing:

1. It works on a package that needs aa to extract
2. What happens if it fails entirely

[grahampugh]

Here's a successful output:

FlatPkgUnpacker
{'Input': {'destination_path': '/Users/gpugh/Library/AutoPkg/Cache/com.github.eth-its-recipes.pkg.TeamViewer/unpack',
           'flat_pkg_path': '/Users/gpugh/Library/AutoPkg/Cache/com.github.eth-its-recipes.pkg.TeamViewer/downloads/TeamViewer.dmg/Install '
                            'TeamViewer.app/Contents/Resources/Install '
                            'TeamViewer.pkg'}}
FlatPkgUnpacker: Mounted disk image /Users/gpugh/Library/AutoPkg/Cache/com.github.eth-its-recipes.pkg.TeamViewer/downloads/TeamViewer.dmg
FlatPkgUnpacker: Unpacked /private/tmp/dmg.PuAkhs/Install TeamViewer.app/Contents/Resources/Install TeamViewer.pkg to /Users/gpugh/Library/AutoPkg/Cache/com.github.eth-its-recipes.pkg.TeamViewer/unpack
{'Output': {}}
PkgPayloadUnpacker
{'Input': {'destination_path': '/Users/gpugh/Library/AutoPkg/Cache/com.github.eth-its-recipes.pkg.TeamViewer/payload',
           'pkg_payload_path': '/Users/gpugh/Library/AutoPkg/Cache/com.github.eth-its-recipes.pkg.TeamViewer/unpack/TeamViewerApp.pkg/Payload'}}
PkgPayloadUnpacker: Unpacked /Users/gpugh/Library/AutoPkg/Cache/com.github.eth-its-recipes.pkg.TeamViewer/unpack/TeamViewerApp.pkg/Payload to /Users/gpugh/Library/AutoPkg/Cache/com.github.eth-its-recipes.pkg.TeamViewer/payload
{'Output': {}}
Versioner
{'Input': {'input_plist_path': '/Users/gpugh/Library/AutoPkg/Cache/com.github.eth-its-recipes.pkg.TeamViewer/payload/TeamViewer.app/Contents/Info.plist',
           'plist_version_key': 'CFBundleShortVersionString'}}
Versioner: No value supplied for skip_single_root_dir, setting default value of: False
Versioner: Found version 15.29.4 in file /Users/gpugh/Library/AutoPkg/Cache/com.github.eth-its-recipes.pkg.TeamViewer/payload/TeamViewer.app/Contents/Info.plist
{'Output': {'version': '15.29.4'}}

Here's a failed output on a Mac that has aa (the failure is caused by a bad pkg_payload_path value):

FlatPkgUnpacker
{'Input': {'destination_path': '/Users/gpugh/Library/AutoPkg/Cache/com.github.eth-its-recipes.pkg.TeamViewer/unpack',
           'flat_pkg_path': '/Users/gpugh/Library/AutoPkg/Cache/com.github.eth-its-recipes.pkg.TeamViewer/downloads/TeamViewer.dmg/Install '
                            'TeamViewer.app/Contents/Resources/Install '
                            'TeamViewer.pkg'}}
FlatPkgUnpacker: Mounted disk image /Users/gpugh/Library/AutoPkg/Cache/com.github.eth-its-recipes.pkg.TeamViewer/downloads/TeamViewer.dmg
FlatPkgUnpacker: Unpacked /private/tmp/dmg.tG1NTL/Install TeamViewer.app/Contents/Resources/Install TeamViewer.pkg to /Users/gpugh/Library/AutoPkg/Cache/com.github.eth-its-recipes.pkg.TeamViewer/unpack
{'Output': {}}
PkgPayloadUnpacker
{'Input': {'destination_path': '/Users/gpugh/Library/AutoPkg/Cache/com.github.eth-its-recipes.pkg.TeamViewer/payload',
           'pkg_payload_path': '/Users/gpugh/Library/AutoPkg/Cache/com.github.eth-its-recipes.pkg.TeamViewer/unpack/TeamViewerApp.pkg/NoPayload'}}
Traceback (most recent call last):
  File "/Library/AutoPkg/autopkglib/__init__.py", line 1016, in process
    self.env = processor.process()
  File "/Library/AutoPkg/autopkglib/__init__.py", line 802, in process
    self.main()
  File "/Library/AutoPkg/autopkglib/PkgPayloadUnpacker.py", line 129, in main
    self.unpack_pkg_payload()
  File "/Library/AutoPkg/autopkglib/PkgPayloadUnpacker.py", line 122, in unpack_pkg_payload
    raise ProcessorError(error)
autopkglib.ProcessorError: extraction of /Users/gpugh/Library/AutoPkg/Cache/com.github.eth-its-recipes.pkg.TeamViewer/unpack/TeamViewerApp.pkg/NoPayload with aa failed
  File "/Library/AutoPkg/autopkglib/__init__.py", line 1016, in process
    self.env = processor.process()
extraction of /Users/gpugh/Library/AutoPkg/Cache/com.github.eth-its-recipes.pkg.TeamViewer/unpack/TeamViewerApp.pkg/NoPayload with aa failed
Failed.

I don't currently have access to the device running Catalina or older to check the output when aa is not present.

[grahampugh]

PS here's the output using GoogleDrive.pkg, which is a CPIO archive Payload. Hint: it's the same output, but ditto is used because it still works on this type of Payload:

{'Output': {}}
FlatPkgUnpacker
{'Input': {'destination_path': '/Users/gpugh/Library/AutoPkg/Cache/com.github.eth-its-recipes.pkg.GoogleDrive/unpack',
           'flat_pkg_path': '/Users/gpugh/Library/AutoPkg/Cache/com.github.eth-its-recipes.pkg.GoogleDrive/downloads/GoogleDrive.dmg/GoogleDrive.pkg'}}
FlatPkgUnpacker: Mounted disk image /Users/gpugh/Library/AutoPkg/Cache/com.github.eth-its-recipes.pkg.GoogleDrive/downloads/GoogleDrive.dmg
FlatPkgUnpacker: Unpacked /private/tmp/dmg.JrcAHf/GoogleDrive.pkg to /Users/gpugh/Library/AutoPkg/Cache/com.github.eth-its-recipes.pkg.GoogleDrive/unpack
{'Output': {}}
PkgPayloadUnpacker
{'Input': {'destination_path': '/Users/gpugh/Library/AutoPkg/Cache/com.github.eth-its-recipes.pkg.GoogleDrive/payload',
           'pkg_payload_path': '/Users/gpugh/Library/AutoPkg/Cache/com.github.eth-its-recipes.pkg.GoogleDrive/unpack/GoogleDrive_x86_64.pkg/Payload'}}
PkgPayloadUnpacker: Unpacked /Users/gpugh/Library/AutoPkg/Cache/com.github.eth-its-recipes.pkg.GoogleDrive/unpack/GoogleDrive_x86_64.pkg/Payload to /Users/gpugh/Library/AutoPkg/Cache/com.github.eth-its-recipes.pkg.GoogleDrive/payload
{'Output': {}}
PlistReader
{'Input': {'info_path': '/Users/gpugh/Library/AutoPkg/Cache/com.github.eth-its-recipes.pkg.GoogleDrive/payload/Google '
                        'Drive.app/Contents/Info.plist',
           'plist_keys': {'CFBundleShortVersionString': 'version',
                          'LSMinimumSystemVersion': 'min_os_version'}}}
PlistReader: Reading: /Users/gpugh/Library/AutoPkg/Cache/com.github.eth-its-recipes.pkg.GoogleDrive/payload/Google Drive.app/Contents/Info.plist
PlistReader: Assigning value of '57.0' to output variable 'version'
PlistReader: Assigning value of '10.13' to output variable 'min_os_version'
{'Output': {'plist_reader_output_variables': {'min_os_version': '10.13',
                                              'version': '57.0'}}}

[colorenz]

Is it possible that we get a Prerelease with that fix? :)

[grahampugh]

@colorenz while we wait, I've created a clone of my fix at com.github.eth-its-recipes.processors/PkgPayloadUnpacker2 and am using it successfully with our own TeamViewer recipes. To use this you would have to create your own TeamViewer.pkg recipe, and autopkg repo-add eth-its/autopkg-mac-recipes-yaml.

Alternatively you can copy the PkgPayloadUnpacker directly from https://github.com/eth-its/autopkg/blob/master/Code/autopkglib/PkgPayloadUnpacker.py into /Library/AutoPkg/autopkglib

[grahampugh]

It looks like somebody else also created a custom processor to switch to aa specifically for TeamViewer: autopkg/hjuutilainen-recipes#221 which has now been merged. Still, it would be good to get this processor updated so we don't need the custom processor.

[nmcspadden]

@grahampugh Move this to :dev and I'll merge.

[grahampugh]

@nmcspadden done.