Unzip failed with interactive message #109

eseliger · 2020-02-12T20:59:31Z

unzip version: UnZip 6.00 of 20 April 2009, by Info-ZIP. Maintained by C. Spieler.
OS: macOS Catalina 10.15.3
Error log:

Unzipping the ZIP archive failed: unzip failed: exit status 1: replace /tmp/action-sd9-akita694954415/angular/playground/projects/ng-entity-service/src/lib/ng-entity-service-notifier.ts? [y]es, [n]o, [A]ll, [N]one, [r]ename:  NULL
(EOF or read error, treating as "[N]one" ...)

# Repository github.com/sd9/akita @ refs/heads/master (1 steps)
# ERROR: Unzipping the ZIP archive failed: unzip failed: exit status 1: replace /tmp/action-sd9-akita391331449/angular/playground/projects/ng-entity-service/src/lib/ng-entity-service-notifier.ts? [y]es, [n]o, [A]ll, [N]one, [r]ename:  NULL
(EOF or read error, treating as "[N]one" ...)

action file:

{
  "scopeQuery": "repo:github.com/sd9/* repohasfile:^package.json",
  "steps": [
    {
      "type": "command",
      "args": ["sed", "-i", "", "s/\"main\"/\"es2015\"/", "package.json"]
    }
  ]
}

The text was updated successfully, but these errors were encountered:

eseliger · 2020-02-12T21:02:31Z

I just saw some other errors popping up:

wechat_web_devtools

# Repository github.com/sd9/wechat_web_devtools @ refs/heads/master (1 steps)
# ERROR: Unzipping the ZIP archive failed: unzip failed: exit status 50: checkdir error:  cannot create /tmp/action-sd9-wechat_web_devtools742807097/Tencent/ե��+��+�������������
                 Illegal byte sequence
                 unable to process Tencent/ե��+��+�������������/.
error:  cannot create /tmp/action-sd9-wechat_web_devtools742807097/images/ե��+����������.jpeg
        Illegal byte sequence
error:  cannot create /tmp/action-sd9-wechat_web_devtools742807097/images/�����1.png
        Illegal byte sequence
error:  cannot create /tmp/action-sd9-wechat_web_devtools742807097/images/ަ�޻�������.png
        Illegal byte sequence
error:  cannot create /tmp/action-sd9-wechat_web_devtools742807097/package.nw/node_modules/git-utils/deps/libgit2/tests/resources/status/�+�
        Illegal byte sequence
error:  cannot create /tmp/action-sd9-wechat_web_devtools742807097/package.nw/node_modules/nodegit/vendor/libgit2/tests/resources/status/�+�
        Illegal byte sequence

(I assume there are chinese characters or something in the filenames)

meedu

# Repository github.com/sd9/meedu @ refs/heads/master (1 steps)
# ERROR: Unzipping the ZIP archive failed: unzip failed: exit status 50: error:  cannot create /tmp/action-sd9-meedu946649789/docs/api/v1/�+����������.md
        Illegal byte sequence
error:  cannot create /tmp/action-sd9-meedu946649789/docs/api/v1/�+����.md
        Illegal byte sequence
error:  cannot create /tmp/action-sd9-meedu946649789/docs/api/v1/�����������.md
        Illegal byte sequence
error:  cannot create /tmp/action-sd9-meedu946649789/docs/api/v1/��+բ������.md
        Illegal byte sequence
error:  cannot create /tmp/action-sd9-meedu946649789/docs/api/v1/޺���������.md
        Illegal byte sequence
error:  cannot create /tmp/action-sd9-meedu946649789/docs/api/v1/޻���������.md
        Illegal byte sequence
error:  cannot create /tmp/action-sd9-meedu946649789/docs/ի���������.md
        Illegal byte sequence
error:  cannot create /tmp/action-sd9-meedu946649789/docs/իܵ������+�.md
        Illegal byte sequence
error:  cannot create /tmp/action-sd9-meedu946649789/docs/���ަ�޺����.md
        Illegal byte sequence
error:  cannot create /tmp/action-sd9-meedu946649789/docs/���޻�������.md
        Illegal byte sequence

react-quill, sourcegraph, sourcegraph-1

# Repository github.com/sd9/react-quill @ refs/heads/master (1 steps)
# Step 0: command [sed -i  s/"main"/"es2015"/ package.json]
# Step 0: done.
# ERROR: Generating a diff failed: diff (output was: "diff: /tmp/action-sd9-react-quill217582422/demo/prop-types.js: No such file or directory\ndiff: /tmp/action-sd9-react-quill257717011/demo/prop-types.js: No such file or directory\ndiff: /tmp/action-sd9-react-quill217582422/demo/quill.snow.css: No such ..."): exit status 2

# Repository github.com/sd9/sourcegraph @ refs/heads/master (1 steps)
# Step 0: command [sed -i  s/"main"/"es2015"/ package.json]
# Step 0: done.
# ERROR: Generating a diff failed: diff (output was: "diff: /tmp/action-sd9-sourcegraph155004138/browser/node_modules/.bin: No such file or directory\ndiff: /tmp/action-sd9-sourcegraph206286072/browser/node_modules/.bin: No such file or directory\ndiff: /tmp/action-sd9-sourcegraph155004138/client/browser/..."): exit status 2

# Repository github.com/sd9/sourcegraph-1 @ refs/heads/master (1 steps)
# Step 0: command [sed -i  s/"main"/"es2015"/ package.json]
# Step 0: done.
# ERROR: Generating a diff failed: diff (output was: "diff: /tmp/action-sd9-sourcegraph-1001303361/browser/node_modules/.bin: No such file or directory\ndiff: /tmp/action-sd9-sourcegraph-1256487927/browser/node_modules/.bin: No such file or directory\ndiff: /tmp/action-sd9-sourcegraph-1001303361/client/br..."): exit status 2

eseliger · 2020-02-12T23:07:15Z

Update: the issue with wechat and meedu is only on macOS, running in linux doesn't yield this problem. Apparently, it's due to APFS:
CocoaPods/CocoaPods#7711

eseliger · 2020-02-12T23:26:45Z

Update: the react-quill, sourcegraph-1 and sourcegraph repos failed because macOS doesn't support the --no-dereference flag. todo should we make the lack of this flag more prominent when such an error occurs? 🤔

(brew install diffutils installs a recent enough version)

slimsag · 2020-02-13T01:22:21Z

Go has a really nice zip package, why aren't we using that instead of shelling out to unzip? https://sourcegraph.com/github.com/sourcegraph/src-cli/-/blob/cmd/src/actions_exec_backend_runner.go#L311

mrnugget · 2020-02-13T08:13:03Z

Update: the react-quill, sourcegraph-1 and sourcegraph repos failed because macOS doesn't support the --no-dereference flag. todo should we make the lack of this flag more prominent when such an error occurs? 🤔

Yeah, the version on macOS is incredibly outdated. We can print a warning in the logs if we detect that the version doesn't support the flags we need.

Go has a really nice zip package, why aren't we using that instead of shelling out to unzip?

No hard reason. If I had to guess I'd say it was easier to use?

mrnugget · 2020-02-13T08:42:24Z

Update: the issue with wechat and meedu is only on macOS, running in linux doesn't yield this problem. Apparently, it's due to APFS:
CocoaPods/CocoaPods#7711

Yeah, there's another issue with APFS: it's case-insensitive by default, whereas most Linux filesystems are case-sensitive by default. And, guess what, some repos have a readme.md and a README.md. That breaks on macOS.

mrnugget · 2020-02-20T10:08:26Z

@eseliger Which of the issues you reported here were fixed by installing a newer version of diff, which ones are due to APFS and which ones are independent and can be fixed by us?

eseliger · 2020-02-20T13:13:52Z

the issues related to react-quill, sourcegraph, sourcegraph-1 are fixed by updating diff as they had softlinks to files that aren't present in the repo

eseliger · 2020-02-20T13:14:54Z

Leftover are the issue originially posted with the interactive prompt from unzip and the issue with the chinese characters (which is a little weird because macOS's internal unzipper tool from finder can unzip it correctly and the folder has the correct name)

mrnugget · 2020-02-20T13:45:33Z

So the issues left are:

Unzipping ZIP archives that include files with the same case-independent name fails on macOS because APFS is case-insensitive:

2. Unzipping files with chinese characters in the name. Interestingly enough, GitHub's own "Download ZIP" functionality gives me a ZIP file that does **not** include those files:

$ find docs
docs
docs/.gitignore
docs/api
docs/api/v1
docs/api/v1/VIP.md
docs/api/v1/README.md
docs/api/.DS_Store

vs.

eseliger · 2020-02-20T15:13:10Z

Ah yep the first makes sense, my bad I didn't see that on first sight.

The latter, that is weird behavior from GitHubs side I think 😀 Given that it's not impossible with APFS, as the internal finder tool can correctly unzip it, maybe we should give the go zip library a try and even get around that dependency on unzip? 🤔 (not sure that can handle it better though, but might be worth a spike)

mrnugget · 2020-02-20T15:21:39Z

maybe we should give the go zip library a try and even get around that dependency on unzip? 🤔 (not sure that can handle it better though, but might be worth a spike)

Yep, I've got it working locally. Will push a commit/PR later today.

This fixes #109 by using Go's archive/zip package to unzip files. (The code is taken from here and slightly modified: https://golangcode.com/unzip-files-in-go/) It fixes the "interactive" problem, because it silently ignores case-sensitive naming conflicts, which is what you get when using macOS' Finder's unzip utility and unzip on the CLI. "Silently ignores" sounds bad but I think it's what you expect to happen on macOS. It also handles the chinese characters correctly.

* Use Go's archive/zip to unzip repository archives This fixes #109 by using Go's archive/zip package to unzip files. (The code is taken from here and slightly modified: https://golangcode.com/unzip-files-in-go/) It fixes the "interactive" problem, because it silently ignores case-sensitive naming conflicts, which is what you get when using macOS' Finder's unzip utility and unzip on the CLI. "Silently ignores" sounds bad but I think it's what you expect to happen on macOS. It also handles the chinese characters correctly. * Be more defensive when unzipping files * Incorporate more feedback

eseliger added the bug Something isn't working label Feb 12, 2020

eseliger assigned mrnugget Feb 12, 2020

mrnugget added the action-exec Everything related to the action exec functionality label Feb 13, 2020

mrnugget mentioned this issue Feb 18, 2020

campaigns: Improve UX of using the src CLI with Automation sourcegraph/sourcegraph-public-snapshot#8479

Closed

13 tasks

mrnugget mentioned this issue Feb 20, 2020

Use Go's archive/zip to unzip repository archives #138

Merged

mrnugget closed this as completed in #138 Feb 20, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unzip failed with interactive message #109

Unzip failed with interactive message #109

eseliger commented Feb 12, 2020 •

edited

Loading

eseliger commented Feb 12, 2020 •

edited

Loading

eseliger commented Feb 12, 2020 •

edited

Loading

eseliger commented Feb 12, 2020 •

edited

Loading

slimsag commented Feb 13, 2020 •

edited

Loading

mrnugget commented Feb 13, 2020

mrnugget commented Feb 13, 2020

mrnugget commented Feb 20, 2020

eseliger commented Feb 20, 2020

eseliger commented Feb 20, 2020

mrnugget commented Feb 20, 2020 •

edited

Loading

eseliger commented Feb 20, 2020

mrnugget commented Feb 20, 2020

Unzip failed with interactive message #109

Unzip failed with interactive message #109

Comments

eseliger commented Feb 12, 2020 • edited Loading

eseliger commented Feb 12, 2020 • edited Loading

eseliger commented Feb 12, 2020 • edited Loading

eseliger commented Feb 12, 2020 • edited Loading

slimsag commented Feb 13, 2020 • edited Loading

mrnugget commented Feb 13, 2020

mrnugget commented Feb 13, 2020

mrnugget commented Feb 20, 2020

eseliger commented Feb 20, 2020

eseliger commented Feb 20, 2020

mrnugget commented Feb 20, 2020 • edited Loading

eseliger commented Feb 20, 2020

mrnugget commented Feb 20, 2020

eseliger commented Feb 12, 2020 •

edited

Loading

eseliger commented Feb 12, 2020 •

edited

Loading

eseliger commented Feb 12, 2020 •

edited

Loading

eseliger commented Feb 12, 2020 •

edited

Loading

slimsag commented Feb 13, 2020 •

edited

Loading

mrnugget commented Feb 20, 2020 •

edited

Loading