failing deploys because of awk: write failure

[julianlconnor]

Hey Heroku,

We began receiving awk: write failure errors this morning at our webpack -p step. We are aware that we should be specifying a buildpack version and will make that change right now.

remote:        Condition always false [./~/transform-loader?envify!./~/react/lib/ReactOwner.js:91,0]
remote:        Condition always false [./~/transform-loader?envify!./~/react/lib/ReactServerRendering.js:29,0]
remote:        Condition always false [./~/transform-loader?envify!./~/react/lib/ReactServerRendering.js:56,0]
remote:        Condition always false [./~/transform-loader?envify!./~/react/lib/SimpleEventPlugin.js:307,0]
remote:
remote: awk: write failure
remote:
remote: -----> Build failed
remote:
remote:        We're sorry this build is failing! You can troubleshoot common issues here:
remote:        https://devcenter.heroku.com/articles/troubleshooting-node-deploys

Nonetheless, others may be encountering this issue. Hope this helps.

[hunterloftis]

Thanks @Muffs! I deployed the awk change because sed was throwing the same errors with large blocks of non-buffered output. We had reports of customers getting better results with awk, but ultimately I think we're going to have to go deeper.

These errors have started to appear recently; they tend to happen with the latest npm and with webpack, both of which yield verbose output. The quick-fix is to minimize output by configuring either npm or webpack to be quieter. Long-term we're looking into whether or not any platform changes could be limiting stdout.

[julianlconnor]

Good call, we'll figure out a way to minimize our webpack output. 😄

[FELS-Rohit]

I am facing same error
remote: readable-stream@1.0.33 /tmp/build_39ca6447318aaa1710723cd2555982c
4/node_modules/loopback-component-storage/node_modules/pkgcloud/node_modules/thr
ough2/node_modules/readable-stream
remote:
remote: awk: write failure
remote:
remote: -----> Build failed
remote:
remote: We're sorry this build is failing! You can troubleshoot common is
sues here:
remote: https://devcenter.heroku.com/articles/troubleshooting-node-deploy
s
remote:
remote: Some possible problems:
remote:
remote: - node_modules checked into source control
remote: https://docs.npmjs.com/misc/faq#should-i-check-my-node-modules-

[hunterloftis]

@FELS-Rohit @Muffs would you try this branch?

#274

heroku buildpacks:set http://github.com/heroku/heroku-buildpack-nodejs#remove-stdout-formatting

I'm interested to see if removing formatting altogether, with either sed or awk, in fact helps your builds.

[FELS-Rohit]

@hunterloftis Thanks For Reply
I had set the build pack but stil have "write error"

remote: readable-stream@1.0.33 /tmp/build_dbf888fac8fa0e62dbd7119f283cbd87/node_modules/loopback-component-stotee: write error remote:

[hunterloftis]

Thanks @FELS-Rohit, that's very helpful. It could be that the tee I've added to output is causing the issue instead; I'll work up another patch.

[antonholmquist]

I'm also seeing this error:

remote: readable-stream@1.1.13 /tmp/build_ecfab0afa1d8f61247cb8d56215230/node_modules/gulp-concat/node_modules/gulp-util/node_modules/multipipe/node_modules/duplexer2/node_modules/readable-stream
remote: awk: write failure

Thankful for a fix or workaround!

[hunterloftis]

@antonholmquist you can probably lock your version to #v81:

heroku buildpacks:set http://github.com/heroku/heroku-buildpack-nodejs#v81

It looks like Ubuntu's tee is missing the option to ignore pipe errors, so I may have to remove it altogether.

[hunterloftis]

@FELS-Rohit okay, give it another go if you can. Also, if anyone is running into this and has an open-source project that consistently gives this issue, please let me know (so I can test it myself). So far I've been unable to reproduce the issue despite trying to simulate dumping a ton of output, and I hate asking you folks to test it repeatedly.

• https://github.com/hunterloftis/test-fast-build-output

[antonholmquist]

Okay, I tried five times straight one hour ago and all failed. A few hours before that it failed the first time and passed right after. Now, when setting the buildback you suggested, it worked the first time.

[FELS-Rohit]

@hunterloftis I have removed the buildpacks stuff and test again
have same error
remote:
remote: awk: write failure (Resource temporarily unavailable)

[FELS-Rohit]

@hunterloftis this is working fine with build pack
$ heroku buildpacks:set http://github.com/heroku/heroku-buildpack-nodejs#fully-
disable-cache

[hunterloftis]

Thanks for the help testing everyone. Because of your input I've been able to reproduce the issue and write a patch. The output function becomes somewhat more complex to handle all the cases; bash experts are welcome to help me simplify these buffering concerns:

# format output on both OSX and Ubuntu, cedar-10 and cedar-14, while also splitting to a log file
output() {
  local logfile="$1"
  local c='s/^/       /'

  if [ "${STACK:-cedar-14}" = "cedar" ]; then
    tee -a "$logfile" | awk -W interactive '{ print "       " $0 }'
  else
    case $(uname) in
      Darwin) sed -l "$c" | tee -i -a "$logfile" 2> /dev/null;;
      *)      stdbuf -oL -eL sed -u "$c" | tee -i -a "$logfile" 2> /dev/null;;
    esac
  fi
}

[FELS-Rohit]

@hunterloftis Will review in morning its very late at my end
Thanks for support

[hunterloftis]

This should be fixed in #v85; please let me know if you're still seeing it.

[trinonsense]

i used #v85 and i'm still seeing the issue, but it doesn't even print write failure anymore. it just fails.

 once@1.3.2 /tmp/build_a9bf6b6f0a165733ad9c22b82a3cf054/node_modules/gulp/node_modules/orchestrator/node_modules/end-of-stream/node_modules/once
       wrappy@1.0.1 /tmp/build_a9bf6b6f0a165733ad9c22b82a3cf054/node_modules/gulp/node_modules/orchestrator/node_modules/end-of-stream/node_modules/once/node_modules/wrappy
       sequenci
-----> Build failed

       We're sorry this build is failing! You can troubleshoot common issues here:
       https://devcenter.heroku.com/articles/troubleshooting-node-deploys

       Some possible problems:

       - node_modules checked into source control
         https://docs.npmjs.com/misc/faq#should-i-check-my-node-modules-folder-into-git

       - Node version not specified in package.json
         https://devcenter.heroku.com/articles/nodejs-support#specifying-a-node-js-version

       Love,
       Heroku


 !     Push rejected, failed to compile Node.js app

[hunterloftis]

You're right @trinonsense, at a certain point (around 4000+ unbuffered stdout writes) it would still fail.

I've just ripped out sed/awk/tee etc altogether and replaced them with a much simpler read/echo loop.