Gradle build daemon disappeared unexpectedly

[rgarrigue]

Hi beevelop

I tried your docker on my workstation to build a ionic2 project. Everything went fine.

Then I created a Gitlab pipeline to do exactly the same thing for our CI/CD... and it fails with a "Gradle build daemon disappeared unexpectedly (it may have been killed or may have crashed)". Tried to get more logs with --debug and --stacktrace, I had no better clue, tried to upgrade the CI's runner VM to 2cpu / 4go ram, not better. Does that ring a bell to you ?

Here's the build stage script

npm install
ionic config set -g telemetry false
ionic cordova platform add android
ionic cordova build android --debug

And here's the log

Error: /builds/trainee/ionic_agile/platforms/android/gradlew: Command failed with exit code 1 Error output:
The message received from the daemon indicates that the daemon has disappeared.
Build request sent: Build{id=615e5a73-f03e-4964-aabf-2c66cffe0aaf.1, currentDir=/builds/trainee/ionic_agile}
Attempting to read last messages from the daemon log...
Daemon pid: 241
  log file: /root/.gradle/daemon/4.1/daemon-241.out.log
----- Last  20 lines from daemon log file - daemon-241.out.log -----
09:25:58.818 [DEBUG] [org.gradle.launcher.daemon.bootstrap.DaemonStartupCommunication] Completed writing the daemon greeting. Closing streams...
09:25:58.823 [DEBUG] [org.gradle.launcher.daemon.server.Daemon] stopOnExpiration() called on daemon
09:25:58.826 [DEBUG] [org.gradle.launcher.daemon.server.Daemon] awaitExpiration() called on daemon
09:25:58.826 [DEBUG] [org.gradle.launcher.daemon.server.DaemonStateCoordinator] daemon is running. Sleeping until state changes.
09:25:58.831 [DEBUG] [org.gradle.internal.remote.internal.inet.TcpIncomingConnector] Accepted connection from /127.0.0.1:42360 to /127.0.0.1:42895.
09:25:59.035 [DEBUG] [org.gradle.launcher.daemon.server.SynchronizedDispatchConnection] thread 14: received class org.gradle.launcher.daemon.protocol.Build
09:25:59.036 [DEBUG] [org.gradle.launcher.daemon.server.DefaultDaemonConnection] thread 14: Received non-IO message from client: Build{id=615e5a73-f03e-4964-aabf-2c66cffe0aaf.1, currentDir=/builds/trainee/ionic_agile}
09:25:59.043 [INFO] [org.gradle.launcher.daemon.server.DefaultIncomingConnectionHandler] Received command: Build{id=615e5a73-f03e-4964-aabf-2c66cffe0aaf.1, currentDir=/builds/trainee/ionic_agile}.
09:25:59.043 [DEBUG] [org.gradle.launcher.daemon.server.DefaultIncomingConnectionHandler] Starting executing command: Build{id=615e5a73-f03e-4964-aabf-2c66cffe0aaf.1, currentDir=/builds/trainee/ionic_agile} with connection: socket connection from /127.0.0.1:42895 to /127.0.0.1:42360.
09:25:59.045 [ERROR] [org.gradle.launcher.daemon.server.DaemonStateCoordinator] Command execution: started DaemonCommandExecution[command = Build{id=615e5a73-f03e-4964-aabf-2c66cffe0aaf.1, currentDir=/builds/trainee/ionic_agile}, connection = DefaultDaemonConnection: socket connection from /127.0.0.1:42895 to /127.0.0.1:42360] after 0.0 minutes of idle
09:25:59.046 [INFO] [org.gradle.launcher.daemon.server.DaemonRegistryUpdater] Marking the daemon as busy, address: [f6bf8e08-2699-4479-8fb8-b0bcac9b5aef port:42895, addresses:[/127.0.0.1]]
09:25:59.046 [DEBUG] [org.gradle.launcher.daemon.registry.PersistentDaemonRegistry] Marking busy by address: [f6bf8e08-2699-4479-8fb8-b0bcac9b5aef port:42895, addresses:[/127.0.0.1]]
09:25:59.046 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Waiting to acquire exclusive lock on daemon addresses registry.
09:25:59.047 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Lock acquired on daemon addresses registry.
09:25:59.050 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
09:25:59.051 [DEBUG] [org.gradle.launcher.daemon.server.DaemonStateCoordinator] resetting idle timer
09:25:59.051 [DEBUG] [org.gradle.launcher.daemon.server.DaemonStateCoordinator] daemon is running. Sleeping until state changes.
09:25:59.055 [INFO] [org.gradle.launcher.daemon.server.exec.StartBuildOrRespondWithBusy] Daemon is about to start building Build{id=615e5a73-f03e-4964-aabf-2c66cffe0aaf.1, currentDir=/builds/trainee/ionic_agile}. Dispatching build started information...
09:25:59.056 [DEBUG] [org.gradle.launcher.daemon.server.SynchronizedDispatchConnection] thread 16: dispatching class org.gradle.launcher.daemon.protocol.BuildStarted
09:25:59.076 [DEBUG] [org.gradle.launcher.daemon.server.exec.EstablishBuildEnvironment] Configuring env variables: {PATH=/builds/trainee/ionic_agile/node_modules/.bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/android/tools:/opt/android/platform-tools:/opt/android/build-tools/27.0.0:/usr/share/ant/bin:/usr/share/maven/bin:/usr/share/gradle/bin:/opt/node/bin:/opt/android/tools/bin, IONIC_SOURCE_MAP_TYPE=source-map, IONIC_CLI_LIB=/opt/node/lib/node_modules/ionic/dist/index.js, IONIC_SELECT_POPOVER_CLASSNAME=SelectPopover, IONIC_PICKER_COMPONENT_PATH=/builds/trainee/ionic_agile/node_modules/ionic-angular/components/picker/picker-component.js, CI_PROJECT_URL=https://gitlab.tools.lagrangenumerique.fr/trainee/ionic_agile, IONIC_APP_SCRIPTS_DIR=/builds/trainee/ionic_agile/node_modules/@ionic/app-scripts, CI_COMMIT_REF_SLUG=master, IONIC_MODAL_CONTROLLER_CLASSNAME=ModalController, CI_PROJECT_PATH_SLUG=trainee-wolly, ANDROID_HOME=/opt/android, IONIC_APP_ENTRY_POINT=/builds/trainee/ionic_agile/src/app/main.ts, IONIC_POPOVER_COMPONENT_FACTORY_PATH=/builds/trainee/ionic_agile/node_modules/ionic-angular/components/popover/popover-component.ngfactory.js, IONIC_MINIFY_CSS=false, IONIC_PLATFORM=android, IONIC_ALERT_COMPONENT_FACTORY_PATH=/builds/trainee/ionic_agile/node_modules/ionic-angular/components/alert/alert-component.ngfactory.js, IONIC_PURGE_UNUSED_FONTS=true, IONIC_MODAL_COMPONENT_FACTORY_PATH=/builds/trainee/ionic_agile/node_modules/ionic-angular/components/modal/modal-component.ngfactory.js, PWD=/builds/trainee/ionic_agile, CI_SERVER=yes, IONIC_LOADING_VIEW_CONTROLLER_PATH=/builds/trainee/ionic_agile/node_modules/ionic-angular/components/loading/loading.js, IONIC_TOAST_VIEW_CONTROLLER_PATH=/builds/trainee/ionic_agile/node_modules/ionic-angular/components/toast/toast.js, IONIC_TMP_DIR=/builds/trainee/ionic_agile/.tmp, CI_BUILD_NAME=build, IONIC_APP_NG_MODULE_CLASS=AppModule, CI_PIPELINE_SOURCE=push, IONIC_LOADING_CONTROLLER_CLASSNAME=LoadingController, CI_RUNNER_ID=18, IONIC_ACTION_SHEET_COMPONENT_PATH=/builds/trainee/ionic_agile/node_modules/ionic-angular/components/action-sheet/action-sheet-component.js, IONIC_ALERT_VIEW_CONTROLLER_PATH=/builds/trainee/ionic_agile/node_modules/ionic-angular/components/alert/alert.js, CI_REGISTRY_USER=gitlab-ci-token, CI_COMMIT_REF_NAME=master, IONIC_ENABLE_LINT=true, IONIC_PIPES_DIR=/builds/trainee/ionic_agile/src/pipes, IONIC_DIRECTIVES_DIR=/builds/trainee/ionic_agile/src/directives, IONIC_RXJS_DIR=/builds/trainee/ionic_agile/node_modules/rxjs, IONIC_ACTION_SHEET_CONTROLLER_CLASSNAME=ActionSheetController, ANDROID_APIS=android-10,android-15,android-16,android-17,android-18,android-19,android-20,android-21,android-22,android-23,android-24,android-25,android-26, IONIC_TOAST_CONTROLLER_PATH=/builds/trainee/ionic_agile/node_modules/ionic-angular/components/toast/toast-controller.js, IONIC_GENERATE_SOURCE_MAP=true, IONIC_MODAL_VIEW_CONTROLLER_PATH=/builds/trainee/ionic_agile/node_modules/ionic-angular/components/modal/modal.js, IONIC_SOURCEMAP_DIR=/builds/trainee/ionic_agile/.sourcemaps, IONIC_POPOVER_VIEW_CONTROLLER_PATH=/builds/trainee/ionic_agile/node_modules/ionic-angular/components/popover/popover.js, SHLVL=1, CI_BUILD_REF_NAME=master, IONIC_NG_MODULE_FILENAME_SUFFIX=.module.ts, IONIC_LOADING_CONTROLLER_PATH=/builds/trainee/ionic_agile/node_modules/ionic-angular/components/loading/loading-controller.js, CI_REGISTRY=dockerhub.tools.lagrangenumerique.fr, IONIC_ROOT_DIR=/builds/trainee/ionic_agile, CI_JOB_STAGE=build, JAVA_HOME=/usr/lib/jvm/java-8-oracle, IONIC_POLYFILL_FILE_NAME=polyfills.js, TERM=xterm, IONIC_READ_CONFIG_JSON=true, IONIC_PROJECT_DIR=/builds/trainee/ionic_agile, IONIC_HTML_TO_SERVE=index.html, GRADLE_HOME=/usr/share/gradle, IONIC_OUTPUT_CSS_FILE_NAME=main.css, ANT_HOME=/usr/share/ant, IONIC_WEBPACK_FACTORY=/builds/trainee/ionic_agile/node_modules/@ionic/app-scripts/dist/webpack/ionic-webpack-factory.js, CI_BUILD_STAGE=build, IONIC_ACTION_SHEET_COMPONENT_FACTORY_PATH=/builds/trainee/ionic_agile/node_modules/ionic-angular/components/action-sheet/action-sheet-component.ngfactory.js, IONIC_TOAST_COMPONENT_PATH=/builds/trainee/ionic_agile/node_modules/ionic-angular/components/toast/toast-component.js, CI_PROJECT_NAMESPACE=trainee, CI_PIPELINE_ID=112, IONIC_ANGULAR_DIR=/builds/trainee/ionic_agile/node_modules/ionic-angular, CI_BUILD_TOKEN=xxxxxxxxxxxxxxxxxxxx, IONIC_ACTION_SHEET_CONTROLLER_PATH=/builds/trainee/ionic_agile/node_modules/ionic-angular/components/action-sheet/action-sheet-controller.js, CI_PROJECT_NAME=ionic_agile, CI_BUILD_REF_SLUG=master, CI=true, IONIC_PROVIDERS_DIR=/builds/trainee/ionic_agile/src/providers, IONIC_PICKER_CONTROLLER_PATH=/builds/trainee/ionic_agile/node_modules/ionic-angular/components/picker/picker-controller.js, IONIC_WEBPACK_LOADER=/builds/trainee/ionic_agile/node_modules/@ionic/app-scripts/dist/webpack/loader.js, CI_JOB_TOKEN=xxxxxxxxxxxxxxxxxxxx, IONIC_APP_NG_MODULE_PATH=/builds/trainee/ionic_agile/src/app/app.module.ts, CI_SERVER_TLS_CA_FILE=/builds/trainee/ionic_agile.tmp/CI_SERVER_TLS_CA_FILE, IONIC_TS_CONFIG=/builds/trainee/ionic_agile/tsconfig.json, IONIC_PICKER_COMPONENT_FACTORY_PATH=/builds/trainee/ionic_agile/node_modules/ionic-angular/components/picker/picker-component.ngfactory.js, IONIC_ANGULAR_TEMPLATE_DIR=/builds/trainee/ionic_agile/node_modules/ionic-angular/templates, IONIC_SELECT_POPOVER_COMPONENT_PATH=/builds/trainee/ionic_agile/node_modules/ionic-angular/components/select/select-popover-component.js, NLSPATH=/usr/dt/lib/nls/msg/%L/%N.cat, CI_PROJECT_DIR=/builds/trainee/ionic_agile, IONIC_MINIFY_JS=false, IONIC_VERSION=3.18.0, IONIC_BUILD_TO_ES5=true, CI_COMMIT_SHA=96afd2eb408285405b6a80914c2e28f75062289a, ANDROID_SDK_URL=https://dl.google.com/android/repository/tools_r25.2.5-linux.zip, IONIC_CLI_BIN=/opt/node/lib/node_modules/ionic/bin/ionic, IONIC_DEEPLINKS_DIR=/builds/trainee/ionic_agile/src, IONIC_SRC_DIR=/builds/trainee/ionic_agile/src, CI_SERVER_REVISION=8895150, IONIC_ENV=dev, IONIC_PARSE_DEEPLINKS=true, IONIC_COMPONENTS_NG_MODULE_PATH=/builds/trainee/ionic_agile/src/components/components.module.ts, IONIC_NODE_MODULES_DIR=/builds/trainee/ionic_agile/node_modules, IONIC_PROJECT_FILE=ionic.config.json, MAVEN_HOME=/usr/share/maven, IONIC_WEBPACK_TRANSPILE_LOADER=/builds/trainee/ionic_agile/node_modules/@ionic/app-scripts/dist/webpack/transpile-loader.js, IONIC_WWW_DIR=/builds/trainee/ionic_agile/www, CI_SERVER_NAME=GitLab, CORDOVA_VERSION=7.1.0, IONIC_ALERT_COMPONENT_PATH=/builds/trainee/ionic_agile/node_modules/ionic-angular/components/alert/alert-component.js, IONIC_BUILD_DIR=/builds/trainee/ionic_agile/www/build, GITLAB_USER_NAME=R??my Garrigue, CI_RUNNER_TAGS=docker, IONIC_MOVE_SOURCE_MAPS=true, IONIC_OPTIMIZATION_LOADER=/builds/trainee/ionic_agile/node_modules/@ionic/app-scripts/dist/webpack/optimization-loader.js, IONIC_COMPONENTS_DIR=/builds/trainee/ionic_agile/src/components, OLDPWD=/builds/trainee/ionic_agile/platforms/android, IONIC_DIRECTIVES_NG_MODULE_PATH=/builds/trainee/ionic_agile/src/directives/directives.module.ts, IONIC_CACHE_LOADER=/builds/trainee/ionic_agile/node_modules/@ionic/app-scripts/dist/webpack/cache-loader.js, ANDROID_BUILD_TOOLS_VERSION=27.0.0, IONIC_LOADING_COMPONENT_FACTORY_PATH=/builds/trainee/ionic_agile/node_modules/ionic-angular/components/loading/loading-component.ngfactory.js, IONIC_PICKER_VIEW_CONTROLLER_PATH=/builds/trainee/ionic_agile/node_modules/ionic-angular/components/picker/picker.js, DEBIAN_FRONTEND=noninteractive, IONIC_FONTS_DIR=/builds/trainee/ionic_agile/www/assets/fonts, GITLAB_CI=true, CI_REGISTRY_IMAGE=dockerhub.tools.lagrangenumerique.fr/trainee/wolly, IONIC_ALERT_CONTROLLER_PATH=/builds/trainee/ionic_agile/node_modules/ionic-angular/components/alert/alert-controller.js, CI_JOB_ID=190, GITLAB_USER_ID=2, IONIC_CLI_LOCAL_ERROR=LOCAL_CLI_NOT_FOUND, IONIC_AOT=false, IONIC_START_WATCH_TIMEOUT=3000, IONIC_POPOVER_CONTROLLER_PATH=/builds/trainee/ionic_agile/node_modules/ionic-angular/components/popover/popover-controller.js, CI_SERVER_VERSION=10.0.3, IONIC_MODAL_COMPONENT_PATH=/builds/trainee/ionic_agile/node_modules/ionic-angular/components/modal/modal-component.js, GITLAB_USER_EMAIL=me@company.com, CI_BUILD_REF=96afd2eb408285405b6a80914c2e28f75062289a, CI_PROJECT_ID=80, XFILESEARCHPATH=/usr/dt/app-defaults/%L/Dt, CI_RUNNER_DESCRIPTION=gitlab-r-02, IONIC_PAGES_DIR=/builds/trainee/ionic_agile/src/pages, CI_REGISTRY_PASSWORD=xxxxxxxxxxxxxxxxxxxx, IONIC_TOAST_CONTROLLER_CLASSNAME=ToastController, CI_PROJECT_PATH=trainee/ionic_agile, IONIC_SELECT_POPOVER_COMPONENT_FACTORY_PATH=/builds/trainee/ionic_agile/node_modules/ionic-angular/components/select/select-popover-component.ngfactory.js, NODEJS_VERSION=8.9.0, IONIC_AT_ANGULAR_DIR=/builds/trainee/ionic_agile/node_modules/@angular, CI_DISPOSABLE_ENVIRONMENT=true, CI_CONFIG_PATH=.gitlab-ci.yml, CI_BUILD_BEFORE_SHA=96afd2eb408285405b6a80914c2e28f75062289a, IONIC_TARGET=cordova, IONIC_APP_ROOT_COMPONENT_PATH=/builds/trainee/ionic_agile/node_modules/ionic-angular/components/app/app-root.js, IONIC_POPOVER_CONTROLLER_CLASSNAME=PopoverController, IONIC_OUTPUT_JS_FILE_NAME=main.js, IONIC_CORE_DIR=/builds/trainee/ionic_agile/node_modules/ionic-angular, IONIC_POPOVER_COMPONENT_PATH=/builds/trainee/ionic_agile/node_modules/ionic-angular/components/popover/popover-component.js, IONIC_CORE_COMPILER_FILE_PATH=/builds/trainee/ionic_agile/node_modules/ionic-angular/compiler, CI_BUILD_ID=190, HOSTNAME=runner-d44571a7-project-80-concurrent-0, CI_JOB_NAME=build, IONIC_OPTIMIZE_JS=false, IONIC_ANGULAR_ENTRY_POINT=/builds/trainee/ionic_agile/node_modules/ionic-angular/index.js, GITLAB_USER_LOGIN=remy.garrigue, IONIC_GLOB_UTIL=/builds/trainee/ionic_agile/node_modules/@ionic/app-scripts/dist/util/glob-util.js, IONIC_LOADING_COMPONENT_PATH=/builds/trainee/ionic_agile/node_modules/ionic-angular/components/loading/loading-component.js, IONIC_PIPES_NG_MODULE_PATH=/builds/trainee/ionic_agile/src/pipes/pipes.module.ts, IONIC_TOAST_COMPONENT_FACTORY_PATH=/builds/trainee/ionic_agile/node_modules/ionic-angular/components/toast/toast-component.ngfactory.js, IONIC_ALERT_CONTROLLER_CLASSNAME=AlertController, CI_REPOSITORY_URL=https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@gitlab.tools.lagrangenumerique.fr/trainee/ionic_agile.git, IONIC_PICKER_CONTROLLER_CLASSNAME=PickerController, HOME=/root, IONIC_ACTION_SHEET_VIEW_CONTROLLER_PATH=/builds/trainee/ionic_agile/node_modules/ionic-angular/components/action-sheet/action-sheet.js, IONIC_MODAL_CONTROLLER_PATH=/builds/trainee/ionic_agile/node_modules/ionic-angular/components/modal/modal-controller.js}
----- End of the daemon log -----


FAILURE: Build failed with an exception.

* What went wrong:
Gradle build daemon disappeared unexpectedly (it may have been killed or may have crashed)

[beevelop]

Did you try either running it without the daemon (--no-daemon or org.gradle.daemon=false in gradle.properties) or upping the memory for the process via org.gradle.jvmargs=-Xmx2048m?

[rgarrigue]

Hi

I've been testing now and then for the last two days. I tweaked memory up, moved the runner to another way more beefy instance, tried multiple gradle daemon options like --no-daemon, latest try being

    - export GRADLE_OPTS="-Xms2048m -Xmx4096m" # Cannot allocate memory
    - ionic cordova build android --debug -- -- --gradleArg="--full-stacktrace --recompile-scripts --refresh-dependencies --rerun-tasks"

No success. It's crashing very fast, that's the only improvement.

[mibzman]

I'm having the same issue. I found some indication here that there might not be a good way to keep the gradle daemon from running. Most of solutions I'm seeing involve upping the machine resources so it doesn't run out of memory, or increasing the heap size.

[beevelop]

@rgarrigue @mibzman Did you guys ever figure out, how to prevent this issue?

[rgarrigue]

Nope, never got it to work, and moved out of the job since then

[beevelop]

Thanks for your update. I'll keep this issue open then until we identified the root cause and a proper workaround to document.

[mark-veenstra]

I encountered the same issues as described here on Gitlab today. I found out that I changed my display name on Gitlab with Emoji's in it. This causes the Gradle to result in an error as described here. So maybe this would help someone identify the same error. Check your display name in profile page or even check all the ENV variables passed to Gradle.

See next bug report: https://gitlab.com/gitlab-org/gitlab-ce/issues/48099

Even if your commit message has an unicode in it, it will break Gradle, see: https://gitlab.com/gitlab-org/gitlab-ce/issues/48099#note_82495926

[mibzman]

I believe my solution was to not run the ionic build portion of my build in Gitlab cloud. I had a local server that I configured to only accept jobs with an ionic-build tag, then assigned that tag to the ionic build step. This worked great since the image stays in the local cache, and more resources were allocated to the build.

[mark-veenstra]

We have a gitlab-runner installed on our own servers and do not use the shared runners. The problem is that unicode in any environment variable used by Gradle just won't work. So avoid any unicode characters in commit messages, titles, display names etc. You can see that initial question here also has this problem. Because the log line org.gradle.launcher.daemon.server.exec.EstablishBuildEnvironment contains a GITLAB_USER_NAME=R??my Garrigue which I guess is the issue. His name contains a unicode character which breaks the Gradle process (see the ??).

EDIT:
Which is correct. See his profile: https://gitlab.tools.lagrangenumerique.fr/remy.garrigue. Because of the unicode character there the build fails. GitLab or Gradle should fix this issue.

[mark-veenstra]

Also see next Gradle issue on GitHub: gradle/gradle#3117 (comment)

[requeijaum]

Hello there, I use GitLab CI and this docker image from beevelop.

Since I branched my repo from "master" branch, I got some weird errors from the pipeline.

I am able to build a release version of my app, but not the debug version - using "--prod release' flag.
But since I don't have a keystore yet: building release versions are useless because Android refuses to install the generated APK.

The debug (normal) build shows that failed because of "Gradle build daemon disappeared unexpectedly".
I will try some tricks mentioned here in this issue in a moment.