Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stdout gets truncated #242

Closed
gajus opened this issue Oct 16, 2021 · 16 comments
Closed

Stdout gets truncated #242

gajus opened this issue Oct 16, 2021 · 16 comments
Labels
feature New feature or request

Comments

@gajus
Copy link

gajus commented Oct 16, 2021

Expected Behavior

Capture the entire output of the program in the result.

Actual Behavior

zx <<'EOF'
await $`unimported`
EOF

Produces result, but cuts off half-way:

02:26:50 ➜ zx <<'EOF'
await $`unimported`
EOF
$ unimported
- initializing

       summary               unimported v1.19.1 (node)
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       entry file 1        : node_modules/@braintree/sanitize-url/src/__tests__
       entry file 2        : node_modules/@braintree/sanitize-url/src/__tests__/test.ts
       entry file 3        : node_modules/@braintree/sanitize-url/src/__tests__/tsconfig.json
       entry file 4        : node_modules/@graphql-tools/relay-operation-optimizer/node_modules/immutable/contrib/cursor/__tests__
       entry file 5        : node_modules/@graphql-tools/relay-operation-optimizer/node_modules/immutable/contrib/cursor/__tests__/Cursor.ts
       entry file 6        : node_modules/@hookform/resolvers/class-validator/src/__tests__
       entry file 7        : node_modules/@hookform/resolvers/class-validator/src/__tests__/__fixtures__
       entry file 8        : node_modules/@hookform/resolvers/class-validator/src/__tests__/__fixtures__/data.ts
       entry file 9        : node_modules/@hookform/resolvers/class-validator/src/__tests__/__snapshots__
       entry file 10       : node_modules/@hookform/resolvers/class-validator/src/__tests__/__snapshots__/class-validator.ts.snap
       entry file 11       : node_modules/@hookform/resolvers/class-validator/src/__tests__/class-validator.ts
       entry file 12       : node_modules/@hookform/resolvers/class-validator/src/__tests__/Form-native-validation.tsx
       entry file 13       : node_modules/@hookform/resolvers/class-validator/src/__tests__/Form.tsx
       entry file 14       : node_modules/@hookform/resolvers/computed-types/src/__tests__
       entry file 15       : node_modules/@hookform/resolvers/computed-types/src/__tests__/__fixtures__
       entry file 16       : node_modules/@hookform/resolvers/computed-types/src/__tests__/__fixtures__/data.ts
       entry file 17       : node_modules/@hookform/resolvers/computed-types/src/__tests__/__snapshots__
       entry file 18       : node_modules/@hookform/resolvers/computed-types/src/__tests__/__snapshots__/computed-types.ts.snap
       entry file 19       : node_modules/@hookform/resolvers/computed-types/src/__tests__/computed-types.ts
       entry file 20       : node_modules/@hookform/resolvers/computed-types/src/__tests__/Form-native-validation.tsx
       entry file 21       : node_modules/@hookform/resolvers/computed-types/src/__tests__/Form.tsx
       entry file 22       : node_modules/@hookform/resolvers/io-ts/src/__tests__
       entry file 23       : node_modules/@hookform/resolvers/io-ts/src/__tests__/__fixtures__
       entry file 24       : node_modules/@hookform/resolvers/io-ts/src/__tests__/__fixtures__/data.ts
       entry file 25       : node_modules/@hookform/resolvers/io-ts/src/__tests__/__snapshots__
       entry file 26       : node_modules/@hookform/resolvers/io-ts/src/__tests__/__snapshots__/io-ts.ts.snap
       entry file 27       : node_modules/@hookform/resolvers/io-ts/src/__tests__/Form-native-validation.tsx
       entry file 28       : node_modules/@hookform/resolvers/io-ts/src/__tests__/Form.tsx
       entry file 29       : node_modules/@hookform/resolvers/io-ts/src/__tests__/io-ts.ts
       entry file 30       : node_modules/@hookform/resolvers/joi/src/__tests__
       entry file 31       : node_modules/@hookform/resolvers/joi/src/__tests__/__fixtures__
       entry file 32       : node_modules/@hookform/resolvers/joi/src/__tests__/__fixtures__/data.ts
       entry file 33       : node_modules/@hookform/resolvers/joi/src/__tests__/__snapshots__
       entry file 34       : node_modules/@hookform/resolvers/joi/src/__tests__/__snapshots__/joi.ts.snap
       entry file 35       : node_modules/@hookform/resolvers/joi/src/__tests__/Form-native-validation.tsx
       entry file 36       : node_modules/@hookform/resolvers/joi/src/__tests__/Form.tsx
       entry file 37       : node_modules/@hookform/resolvers/joi/src/__tests__/joi.ts
       entry file 38       : node_modules/@hookform/resolvers/nope/src/__tests__
       entry file 39       : node_modules/@hookform/resolvers/nope/src/__tests__/__fixtures__
       entry file 40       : node_modules/@hookform/resolvers/nope/src/__tests__/__fixtures__/data.ts
       entry file 41       : node_modules/@hookform/resolvers/nope/src/__tests__/__snapshots__
       entry file 42       : node_modules/@hookform/resolvers/nope/src/__tests__/__snapshots__/nope.ts.snap
       entry file 43       : node_modules/@hookform/resolvers/nope/src/__tests__/Form-native-validation.tsx
       entry file 44       : node_modules/@hookform/resolvers/nope/src/__tests__/Form.tsx
       entry file 45       : node_modules/@hookform/resolvers/nope/src/__tests__/nope.ts
       entry file 46       : node_modules/@hookform/resolvers/superstruct/src/__tests__
       entry file 47       : node_modules/@hookform/resolvers/superstruct/src/__tests__/__fixtures__
       entry file 48       : node_modules/@hookform/resolvers/superstruct/src/__tests__/__fixtures__/data.ts
       entry file 49       : node_modules/@hookform/resolvers/superstruct/src/__tests__/__snapshots__
       entry file 50       : node_modules/@hookform/resolvers/superstruct/src/__tests__/__snapshots__/superstruct.ts.snap
       entry file 51       : node_modules/@hookform/resolvers/superstruct/src/__tests__/Form-native-validation.tsx
       entry file 52       : node_modules/@hookform/resolvers/superstruct/src/__tests__/Form.tsx
       entry file 53       : node_modules/@hookform/resolvers/superstruct/src/__tests__/superstruct.ts
       entry file 54       : node_modules/@hookform/resolvers/typanion/src/__tests__
       entry file 55       : node_modules/@hookform/resolvers/typanion/src/__tests__/__fixtures__
       entry file 56       : node_modules/@hookform/resolvers/typanion/src/__tests__/__fixtures__/data.ts
       entry file 57       : node_modules/@hookform/resolvers/typanion/src/__tests__/__snapshots__
       entry file 58       : node_modules/@hookform/resolvers/typanion/src/__tests__/__snapshots__/typanion.ts.snap
       entry file 59       : node_modules/@hookform/resolvers/typanion/src/__tests__/Form-native-validation.tsx
       entry file 60       : node_modules/@hookform/resolvers/typanion/src/__tests__/Form.tsx
       entry file 61       : node_modules/@hookform/resolvers/typanion/src/__tests__/typanion.ts
       entry file 62       : node_modules/@hookform/resolvers/vest/src/__tests__
       entry file 63       : node_modules/@hookform/resolvers/vest/src/__tests__/__fixtures__
       entry file 64       : node_modules/@hookform/resolvers/vest/src/__tests__/__fixtures__/data.ts
       entry file 65       : node_modules/@hookform/resolvers/vest/src/__tests__/__snapshots__
       entry file 66       : node_modules/@hookform/resolvers/vest/src/__tests__/__snapshots__/vest.ts.snap
       entry file 67       : node_modules/@hookform/resolvers/vest/src/__tests__/Form-native-validation.tsx
       entry file 68       : node_modules/@hookform/resolvers/vest/src/__tests__/Form.tsx
       entry file 69       : node_modules/@hookform/resolvers/vest/src/__tests__/vest.ts
       entry file 70       : node_modules/@hookform/resolvers/yup/src/__tests__
       entry file 71       : node_modules/@hookform/resolvers/yup/src/__tests__/__fixtures__
       entry file 72       : node_modules/@hookform/resolvers/yup/src/__tests__/__fixtures__/data.ts
       entry file 73       : node_modules/@hookform/resolvers/yup/src/__tests__/__snapshots__
       entry file 74       : node_modules/@hookform/resolvers/yup/src/__tests__/__snapshots__/yup.ts.snap
       entry file 75       : node_modules/@hookform/resolvers/yup/src/__tests__/Form-native-validation.tsx
       entry file 76       : node_modules/@hookform/resolvers/yup/src/__tests__/Form.tsx
       entry file 77       : node_modules/@Error: - initializing
    at file:///private/var/folders/j0/ljpkc6ps1kb6x0ts9fmzmx3h0000gn/T/l82xxonpucp.mjs:1:8
    exit code: 1
contra-web-app on master [$] took 57s

Expected behavior

Result of the program is captured in its entirety.

The actual output is a lot longer and ends with a summary:

02:27:53 ➜ unimported

       summary               unimported v1.19.1 (node)
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       entry file 1        : node_modules/@braintree/sanitize-url/src/__tests__
       entry file 2        : node_modules/@braintree/sanitize-url/src/__tests__/test.ts
       entry file 3        : node_modules/@braintree/sanitize-url/src/__tests__/tsconfig.json
       entry file 4        : node_modules/@graphql-tools/relay-operation-optimizer/node_modules/immutable/contrib/cursor/__tests__
       entry file 5        : node_modules/@graphql-tools/relay-operation-optimizer/node_modules/immutable/contrib/cursor/__tests__/Cursor.ts
       entry file 6        : node_modules/@hookform/resolvers/class-validator/src/__tests__
       entry file 7        : node_modules/@hookform/resolvers/class-validator/src/__tests__/__fixtures__
       entry file 8        : node_modules/@hookform/resolvers/class-validator/src/__tests__/__fixtures__/data.ts
       entry file 9        : node_modules/@hookform/resolvers/class-validator/src/__tests__/__snapshots__
       entry file 10       : node_modules/@hookform/resolvers/class-validator/src/__tests__/__snapshots__/class-validator.ts.snap
       entry file 11       : node_modules/@hookform/resolvers/class-validator/src/__tests__/class-validator.ts
       entry file 12       : node_modules/@hookform/resolvers/class-validator/src/__tests__/Form-native-validation.tsx
       entry file 13       : node_modules/@hookform/resolvers/class-validator/src/__tests__/Form.tsx
[trimmed]
─────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
     │ 7 unimported files
─────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1 │ src/components/ContraImage/ContraImage.tsx
   2 │ src/features/profile/components/ProfileSocialLinks/ProfileSocialLinks.tsx
   3 │ src/features/project/components/Project/ProjectExpandedCollaboratorsList/ProjectExpandedCollaboratorsList.tsx
   4 │ src/features/project/components/Project/ProjectMinimizedCollaboratorsList/ProjectMinimizedCollaboratorsList.tsx
   5 │ src/features/project/components/Project/ProjectPartnerWithAction/ProjectPartnerWithAction.tsx
   6 │ src/features/project/components/Project/ProjectRolesToolsOrganizations/ProjectRolesToolsOrganizations.tsx
   7 │ src/features/work/components/Contract/InquirySent/InquirySent.tsx
─────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────


       Inspect the results and run npx unimported -u to update ignore lists

Specifications

@gajus gajus changed the title Does not capture the entire output Produces error when executing Node.js script Oct 16, 2021
@antonmedv
Copy link
Collaborator

Looks like trying to open await file.

Make sure your shell supports heredoc.

@gajus gajus changed the title Produces error when executing Node.js script The output gets cut off Oct 16, 2021
@gajus
Copy link
Author

gajus commented Oct 16, 2021

Oops.. messed up replicating the error. Clarified the issue.

For what it is worth, the only workaround I could think of that works is:

unimported_output=$(mktemp);
unimported > "$unimported_output";
ts-node --transpile-only bin/delete-unimported.ts < "$unimported_output"

@antonmedv
Copy link
Collaborator

Can you create an example so I can debug the issue? For me zx captures everything okay.

@gajus
Copy link
Author

gajus commented Oct 18, 2021

echo 'console.log(
  `Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec non tellus ac eros mattis vehicula. Donec ac hendrerit urna. Suspendisse metus purus, venenatis ut consectetur eu, mattis id orci. Fusce vehicula mattis rutrum. Pellentesque et viverra risus. Donec id odio sit amet nibh maximus fringilla et in nulla. Nulla nec mi sit amet ex aliquet eleifend ac vel ligula. Aenean eleifend elementum turpis, et convallis odio hendrerit at. In quis eros eu tellus laoreet semper efficitur a neque. Pellentesque risus lectus, tempus nec arcu quis, sagittis suscipit turpis. Aliquam fringilla arcu ante. Aliquam varius odio ut nisl hendrerit mollis. Mauris a nisi ligula. Ut dapibus tristique finibus.
`.repeat(100)
);

console.log("end");

process.exit(1);
' > test.js

zx <<'EOF'
await $`node test.js`
EOF

This replicates what's happening with unimported.

I expect the last line of output to be end, instead it is cut off midway.

In contrast, if you just execute node test.js, it will produce the entire output.

@antonmedv
Copy link
Collaborator

antonmedv commented Oct 18, 2021

I think I get it. Calling process.exit(1) will force exit it without waiting.
If we set exitCode, everything works:

process.exitCode = 1;

@gajus
Copy link
Author

gajus commented Oct 18, 2021

That's all good, but I don't get to control how other programs are implemented.

This issue just highlights that zx cannot capture the output of applications that exit this way, where as just running them in bash works.

@antonmedv
Copy link
Collaborator

Looks like it's nodejs isses. See nodejs/node#6379

I was not able to find a solution from the client-side.

As a mitigation: file a bug to affected projects.

@antonmedv
Copy link
Collaborator

But yes, in bash it outputs full text. So maybe there is a workaround. But I don’t know how to do it. ¯\(ツ)

@antonmedv
Copy link
Collaborator

Other languages also see only truncated output:

php -r 'echo `node test.js`;'
ruby -e 'puts `node test.js`'

@gajus
Copy link
Author

gajus commented Oct 18, 2021

I cannot think of a nice workaround, but... you could make it work using the same workaround that I've shared earlier, i.e.

const child = spawn("bash", ["-c", "node print.js > foo"], {
  cwd: __dirname,
});

Where foo is a temporary file and upon completion of the program, you return the contents of foo.

@antonmedv
Copy link
Collaborator

This is not really elegant solution =(

@gajus
Copy link
Author

gajus commented Oct 18, 2021

What's the downside of this workaround?

From user's perspective it gets the job done.

@antonmedv
Copy link
Collaborator

How parallel executions will work?
How real-time output will work?
How this will work on read-only systems?

@gajus
Copy link
Author

gajus commented Oct 18, 2021

How parallel executions will work?
How this will work on read-only systems?

These two shouldn't be an issue using tmp files in memory.

How real-time output will work?

Cannot think of a solution for this though.

@gajus
Copy link
Author

gajus commented Oct 18, 2021

I've been digging through different ways to make this work and found this solution:

const { spawn } = require("child_process");

const child = spawn("script", ["-q", "/dev/null", "node", "./print.js"], {
  cwd: __dirname,
});

child.stdout.on("data", (data) => {
  console.log(String(data));
});

It captures the entire output.

@antonmedv antonmedv added the bug Something isn't working label Jan 16, 2022
@antonmedv antonmedv removed the bug Something isn't working label Feb 16, 2022
@antonmedv antonmedv changed the title The output gets cut off Stdout gets truncated Feb 16, 2022
@antonmedv antonmedv added the feature New feature or request label Feb 16, 2022
@antonmedv
Copy link
Collaborator

antonmedv commented Feb 16, 2022

I wrote a doc on how to deal with this issue: https://google.github.io/zx/known-issues#output-gets-truncated

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants