Azure App Service Deploy failed - JavaScript heap out of memory

[gongdo]

I'm trying to deploy my NodeJS app using Azure App Service Deploy in the Release feature.
I'm currently using VSTS default Hosted VS2017 as an agent.

Here's log:

2017-07-06T08:48:14.6134738Z ##[section]Starting: Azure App Service Deploy: bapul-learning-admin-test
2017-07-06T08:48:14.6274747Z ==============================================================================
2017-07-06T08:48:14.6274747Z Task         : Azure App Service Deploy
2017-07-06T08:48:14.6274747Z Description  : Update Azure App Service using Web Deploy / Kudu REST APIs
2017-07-06T08:48:14.6274747Z Version      : 3.3.9
2017-07-06T08:48:14.6274747Z Author       : Microsoft Corporation
2017-07-06T08:48:14.6274747Z Help         : [More Information](https://aka.ms/azurermwebdeployreadme)
2017-07-06T08:48:14.6274747Z ==============================================================================
2017-07-06T08:48:17.7499766Z Got connection details for Azure App Service:'bapul-learning-admin-test'
2017-07-06T08:48:51.9896729Z FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
2017-07-06T08:48:52.1706751Z ##[error]Exit code 3 returned from process: file name 'C:\LR\mms\Services\Mms\Provisioner\TaskAgent\agents\2.119.1\externals\node\bin\node.exe', arguments '"d:\a\_tasks\AzureRmWebAppDeployment_497d490f-eea7-4f2b-ab94-48d9c1acdcb1\3.3.9\azurermwebappdeployment.js"'.

I don't think that my app causes this failure, since the build process was succeeded without any error.

[vincent1173]

@gongdo , Can you please share the debug logs? (by enabling system.debug = true and requeue the release).

[gongdo]

@vincentdass Ok, here it is.

******************************************************************************
Starting: Azure App Service Deploy: my-nodejs-app
******************************************************************************
==============================================================================
Task         : Azure App Service Deploy
Description  : Update Azure App Service using Web Deploy / Kudu REST APIs
Version      : 3.3.9
Author       : Microsoft Corporation
Help         : [More Information](https://aka.ms/azurermwebdeployreadme)
==============================================================================
agent.workFolder=d:\a
loading inputs and endpoints
loading ENDPOINT_AUTH_ebda3b83-bb72-4b27-8847-b44970e70be8
loading ENDPOINT_AUTH_PARAMETER_ebda3b83-bb72-4b27-8847-b44970e70be8_SERVICEPRINCIPALID
loading ENDPOINT_AUTH_PARAMETER_ebda3b83-bb72-4b27-8847-b44970e70be8_SERVICEPRINCIPALKEY
loading ENDPOINT_AUTH_PARAMETER_ebda3b83-bb72-4b27-8847-b44970e70be8_TENANTID
loading ENDPOINT_AUTH_PARAMETER_SYSTEMVSSCONNECTION_ACCESSTOKEN
loading ENDPOINT_AUTH_SCHEME_ebda3b83-bb72-4b27-8847-b44970e70be8
loading ENDPOINT_AUTH_SCHEME_SYSTEMVSSCONNECTION
loading ENDPOINT_AUTH_SYSTEMVSSCONNECTION
loading INPUT_CONNECTEDSERVICENAME
loading INPUT_DEPLOYTOSLOTFLAG
loading INPUT_EXCLUDEFILESFROMAPPDATAFLAG
loading INPUT_GENERATEWEBCONFIG
loading INPUT_INLINESCRIPT
loading INPUT_PACKAGE
loading INPUT_REMOVEADDITIONALFILESFLAG
loading INPUT_RENAMEFILESFLAG
loading INPUT_SCRIPTPATH
loading INPUT_SERVERFARMID
loading INPUT_SETPARAMETERSFILE
loading INPUT_TAKEAPPOFFLINEFLAG
loading INPUT_USEWEBDEPLOY
loading INPUT_WEBAPPKIND
loading INPUT_WEBAPPNAME
loading INPUT_XMLTRANSFORMATION
loading INPUT_XMLVARIABLESUBSTITUTION
loaded 25
AZURE_HTTP_USER_AGENT=VSTS_86482e0e-8930-425a-92cc-fb01e3cc9228_build_18_349
AZURE_HTTP_USER_AGENT=VSTS_86482e0e-8930-425a-92cc-fb01e3cc9228_build_18_349
check path : d:\a\_tasks\AzureRmWebAppDeployment_497d490f-eea7-4f2b-ab94-48d9c1acdcb1\3.3.9\task.json
set resource file to: d:\a\_tasks\AzureRmWebAppDeployment_497d490f-eea7-4f2b-ab94-48d9c1acdcb1\3.3.9\task.json
system.culture=en-US
ConnectedServiceName=ebda3b83-bb72-4b27-8847-b44970e70be8
WebAppName=my-nodejs-app
DeployToSlotFlag=false
ResourceGroupName=null
SlotName=null
Package=d:\a\1\a\349.zip
VirtualApplication=null
UseWebDeploy=true
SetParametersFile=d:\a\1\s
RemoveAdditionalFilesFlag=false
ExcludeFilesFromAppDataFlag=false
TakeAppOfflineFlag=false
RenameFilesFlag=false
AdditionalArguments=null
WebAppUri=null
XmlTransformation=false
JSONFiles=null
XmlVariableSubstitution=false
ScriptType=null
InlineScript=:: You can provide your deployment commands here. One command per line.
ScriptPath=d:\a\1\s
GenerateWebConfig=false
WebConfigParameters=null
WebAppKind=app
DockerNamespace=null
ebda3b83-bb72-4b27-8847-b44970e70be8 auth param serviceprincipalid = ********
ebda3b83-bb72-4b27-8847-b44970e70be8 auth param serviceprincipalkey = ********
ebda3b83-bb72-4b27-8847-b44970e70be8 auth param tenantid = ********
ebda3b83-bb72-4b27-8847-b44970e70be8 data subscriptionid = ********
ebda3b83-bb72-4b27-8847-b44970e70be8 data environmentAuthorityUrl = https://login.windows.net/
ebda3b83-bb72-4b27-8847-b44970e70be8=https://management.azure.com/
Requesting for Auth Token: https://login.windows.net/********/oauth2/token/
Requesting Azure App Service ID: https://management.azure.com/subscriptions/********/resources?$filter=resourceType EQ 'Microsoft.Web/Sites' AND name EQ 'my-nodejs-app'&api-version=2016-07-01
Web App details : /subscriptions/********/resourceGroups/my-nodejs-app/providers/Microsoft.Web/sites/my-nodejs-app
Azure Resource Group Name : my-nodejs-app
Requesting for Auth Token: https://login.windows.net/********/oauth2/token/
Requesting Azure Publish Profile: https://management.azure.com/subscriptions/********/resourceGroups/my-nodejs-app/providers/Microsoft.Web/sites/my-nodejs-app/publishxml?api-version=2016-08-01
Got connection details for Azure App Service:'my-nodejs-app'
Performing the deployment of webapp.
Finding files matching input: d:\a\1\a\349.zip
Azure web app in wrong state, status code : 301
Failed to check availability of azure web app, error : undefined
Requesting for Auth Token: https://login.windows.net/********/oauth2/token/
Requesting for the Current List of App Settings: https://management.azure.com/subscriptions/********/resourceGroups/my-nodejs-app/providers/Microsoft.Web/sites/my-nodejs-app/config/appsettings/list?api-version=2016-08-01
Processed: ##vso[task.setvariable variable=websiteUserName;issecret=true;]********
Processed: ##vso[task.setvariable variable=websitePassword;issecret=true;]********
SetParametersFile=d:\a\1\s
Absolute path for pathSegments: d:\a\1\s = d:\a\1\s
build.sourcesDirectory=d:\a\1\s
Absolute path for pathSegments: d:\a\1\s = d:\a\1\s
SetParametersFilepath supplied :false
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
Exit code 3 returned from process: file name 'C:\LR\mms\Services\Mms\Provisioner\TaskAgent\agents\2.119.1\externals\node\bin\node.exe', arguments '"d:\a\_tasks\AzureRmWebAppDeployment_497d490f-eea7-4f2b-ab94-48d9c1acdcb1\3.3.9\azurermwebappdeployment.js"'.
Microsoft.VisualStudio.Services.Agent.ProcessExitCodeException: Exit code 3 returned from process: file name 'C:\LR\mms\Services\Mms\Provisioner\TaskAgent\agents\2.119.1\externals\node\bin\node.exe', arguments '"d:\a\_tasks\AzureRmWebAppDeployment_497d490f-eea7-4f2b-ab94-48d9c1acdcb1\3.3.9\azurermwebappdeployment.js"'.
******************************************************************************
Finishing: Azure App Service Deploy: my-nodejs-app
******************************************************************************

Still, I'm not sure what is the problem in this settings.
Deploying by Deployment options in the Azure Web Services works fine with same source code repository.

[gongdo]

I found that this error occurs when the deployment package(.zip) is too large, about 500MB.
Maybe I can split large package into small packages, but I have not tried yet.

[vincent1173]

@gongdo , can you please provide us details on how the package (zip) is generated? (generated from MSBuild / Gulp / Grunt or others).
Can you please share the build logs for the same release ?

[vincent1173]

@gongdo , are you able to deploy the package ? Can you please provide the details for the questions asked above to invesitgate ?

[asranja]

@gongdo, waiting for the package details to debug further.

[cybage-sumedh]

Same error for me. Using Gulp.

[asranja]

Thanks for reporting this @cybage-sumedh . Can you share the debug logs ?

[asranja]

@cybage-sumedh We were looking for debug logs. To get the debug logs add the variable system.debug and set its value to true. Queue the build again.

[cybage-sumedh]

Sure... i will do the same and will post soon

[asranja]

Please send the logs to RM_Customer_Queries at microsoft dot com since this is a public forum.

[gongdo]

@vincentdass @asranja I made fake app that reproduce the problem.
https://github.com/gongdo/BigUselessSample

[added]
https://github.com/gongdo/BigUselessSample/blob/master/test-big-dist.json
Here's the exported build process. I replaced some service names to fake names, so I think it's safe to share it.

As you can see, there's almost nothing.
It just installs bunch of packages and downloads huge resource(font package from Google ;) after npm install(see postinstall command).

And I setup VSTS Build like this:

1. Get sources from the Git repository.
2. npm install
3. Archive files
4. Azure App Service Deploy (the zip from 3)

I'm not sure what exactly makes the JavaScript heap out of memory, but it seems to be related with total size of the contents in the zip file.

I solved this problem to deploy only minified js from the build. But I think many node developers prefer to deploy git repository as it is.

[vincent1173]

@gongdo , thanks for the repro steps, really appreciated.
We have found the root cause of the issue and working on the fix.

Please follow any of the workaround provided below to unblock yourself:

• Remove Archive files task and provide the folder path in "Package or Folder" input of Azure App Service Deploy Task
• Use Post Deployment Script feature in Azure App Service Deploy Task to run your "npm command"

[gongdo]

@vincentdass Your solution is working, but that's what exactly what I wanted to avoid.
If you deploy your Node app with node_modules but not zipping, it would take numerous network round-trips between CI and target.
If you deploy it without node_modules and npm install on the target machine, it would consume (relatively) huge resources on the server and it might affect service performance.

I'm not sure what deploying method would be the best for a Node app on the Azure App Services, but it's definitely out of this topic. It would be a case-by-case solution as always :)

Anyway I'm happy you've found the root cause of the problem.

[AlexanderParadeis]

@vincentdass are you still working on an solution, because i have exactly the same issue with the deployment?

[vincent1173]

@AlexanderParadeis , sorry the issue is not fixed yet.
Can you try the steps provided here as workaround? We are currently working on finding the better fix for the issue. Will update the thread, if there is any updates.
Please share the app type (node/.net).

[AlexanderParadeis]

Thanks for the fast reply, i will try the steps as a workaround.
The app type is a php app with some composer and npm tasks and the whole package size with all modules is around 300mb.

[nickraphael]

I'm getting this error while building an angular app...

...
90% chunk assets processing���������������������������� 90% additional chunk assets processing� �� �� �� �� �� �� �� �� �� �� �� �� �� �� �� �� �� �� �� �� �� �� �� �� ��������������� 91% recording�������������� 91% additional asset processing�������������������������������� 91% additional asset processing scripts-webpack-plugin� �� �� �� �� �� �� �� �� �� �� �� �� �� �� �� �� �� �� �� �� �� �� �� �� �� ������������������������������ 92% chunk asset optimization����������������������������� 92% chunk asset optimization cleancss-webpack-pluginFATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

[GaurangRLB]

Anyone found solution for this. We are using hybrid angular/angular JS application and our build is stuck an ran for 60 minutes and time out. After made changes in build script by using --max_old_space setting as recommended here angular/angular-cli#5618.

[nickraphael]

It's not fixed. Those of us with large enterprise apps cannot fix this with max_old_space.

[GaurangRLB]

Thanks @nickraphael for your reply. It is working fine with --max_old_space on my local PC, but the build on vsts failed.

[iures]

I'm also experiencing the same issues here. Build keeps getting stuck when I use max_old_space.

[vincent1173]

@Pumba-Coder , Please share the task version that you are currently using.
the issue is been fixed in 1.* of new tasks:
Azure Function
Azure Web App

[ghost]

@vincentdass

Working with a hosted vs2017 deployment using version 4.3.33 as seen below:

==============================================================================
2019-07-02T07:28:56.6932747Z Task : Azure App Service deploy
2019-07-02T07:28:56.6932810Z Description : Deploy to Azure App Service a web, mobile, or API app using Docker, Java, .NET, .NET Core, Node.js, PHP, Python, or Ruby
2019-07-02T07:28:56.6932885Z Version : 4.3.33
2019-07-02T07:28:56.6932930Z Author : Microsoft Corporation
2019-07-02T07:28:56.6933005Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/deploy/azure-rm-web-app-deployment
2019-07-02T07:28:56.6933071Z ==============================================================================

agent version is C:\agents\2.153.2\externals\node\bin\node.exe.

[vincent1173]

@Pumba-Coder , Can you try using the new task:

[ghost]

@vincentdass I have changed over to use that task and it will run through fine, but it wont start my server.
Updating the variables to point to my server file using the "Generate web.config parameters" and then running through i will get the out of memory exception again.
Task Version being used is: : 1.0.19

[vincent1173]

@Pumba-Coder , Can you please share the debug logs (set variuable system.debug to true and rerun the release)

[ghost]

@vincentdass I've attached the run agent log file.
1_Run on agent.log

web app deploy:
3_Azure Web App Deploy g-alpha.log

[nitinmb]

@vincentdass - Any update on this issue as even I am noticing the same error as @Pumba-Coder after enabling "Generate web.config" and using new "Azure Web App" task.

[vincent1173]

@nitinmb , looking into this issue. Will update once there is anytrack found.

[nitinmb]

@vincentdass - Do you have any updates on this ?

[zhangwk02]

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory

<--- Last few GCs --->

58445 ms: Mark-sweep 1372.8 (1435.0) -> 1372.7 (1435.0) MB, 1090.0 / 0.0 ms [allocation failure] [GC in old space requested].
59536 ms: Mark-sweep 1372.7 (1435.0) -> 1372.7 (1435.0) MB, 1090.1 / 0.0 ms [allocation failure] [GC in old space requested].
60629 ms: Mark-sweep 1372.7 (1435.0) -> 1374.2 (1407.0) MB, 1093.4 / 0.0 ms [last resort gc].
61735 ms: Mark-sweep 1374.2 (1407.0) -> 1375.7 (1407.0) MB, 1105.0 / 0.0 ms [last resort gc].
1: node::Abort() [/workspaces/externals/node/bin/node]

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x24d893bcfb39
1: realpathSync [fs.js:1667] [pc=0x1e6d22e38d80] (this=0x10641b8cbb9 <an Object with map 0xa0e2ca1a9e9>,p=0x2ff86a26db09 <String[291]: /workspaces/_work/3/s/node_modules/_babel-preset-es2015@6.24.1@babel-preset-es2015/node_modules/babel-plugin-transform-es2015-classes/node_modules/babel-helper-function-name/node_modules/babel-template/node_modules/babel-types/node_modules/babel-runtime/node...

2: 0x109b9cc [/workspaces/externals/node/bin/node]
3: v8::Utils::ReportApiFailure(char const*, char const*) [/workspaces/externals/node/bin/node]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/workspaces/externals/node/bin/node]
5: v8::internal::Factory::NewInternalizedStringImpl(v8::internal::Handlev8::internal::String, int, unsigned int) [/workspaces/externals/node/bin/node]
6: v8::internal::StringTable::LookupString(v8::internal::Isolate*, v8::internal::Handlev8::internal::String) [/workspaces/externals/node/bin/node]
7: v8::internal::LookupIterator::PropertyOrElement(v8::internal::Isolate*, v8::internal::Handlev8::internal::Object, v8::internal::Handlev8::internal::Object, bool*, v8::internal::LookupIterator::Configuration) [/workspaces/externals/node/bin/node]
8: v8::internal::Runtime::GetObjectProperty(v8::internal::Isolate*, v8::internal::Handlev8::internal::Object, v8::internal::Handlev8::internal::Object) [/workspaces/externals/node/bin/node]
9: 0xed3151 [/workspaces/externals/node/bin/node]
10: v8::internal::Runtime_KeyedGetProperty(int, v8::internal::Object**, v8::internal::Isolate*) [/workspaces/externals/node/bin/node]
11: 0x1e6d22d092a7

[bsoulier]

This is not an issue with Azure pipeline per say, as you can override this behavior to run your build with more memory and still get it to succeed.

An updated version on how to do this isto use the npm package CrossEnv and to set it up this way in your package.json in order to run:

"scripts": {
    "build:prod:aot": "cross-env NODE_OPTIONS=--max_old_space_size=8000 ng build --prod --optimization --build-optimizer --aot --progress"
}

Then you just run your script from properly configuring your Azure pipeline task to do so and you are good to go.

[zhangwk02]

ths！ &nbsp; i have resloved the problem by add the scripts&nbsp;yarn upgrade gulp 发自我的iPhone

…

------------------ Original ------------------ From: Benjamin Soulier <notifications@github.com&gt; Date: Tue,Sep 24,2019 11:37 PM To: microsoft/azure-pipelines-tasks <azure-pipelines-tasks@noreply.github.com&gt; Cc: zhangwk02 <415675018@qq.com&gt;, Comment <comment@noreply.github.com&gt; Subject: Re: [microsoft/azure-pipelines-tasks] Azure App Service Deploy failed - JavaScript heap out of memory (#4734) This is not an issue with Azure pipeline per say, as you can override this behavior to run your build with more memory and still get it to succeed. An updated version on how to do this isto use the npm package CrossEnv and to set it up this way in your package.json in order to run: "scripts": { "build:prod:aot": "cross-env NODE_OPTIONS=--max_old_space_size=8000 ng build --prod --optimization --build-optimizer --aot --progress" } Then you just run your script from properly configuring your Azure pipeline task to do so and you are good to go. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[krishnamhn009]

I am facing same issue.I tried to fix using
node --max-old-space-size=8172

it worked for few times and again I am facing same issue.

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory 2019-10-09T08:38:06.4708120Z 2019-10-09T08:38:06.4710954Z <--- Last few GCs ---> 2019-10-09T08:38:06.4711476Z 2019-10-09T08:38:06.4712223Z 36339 ms: Mark-sweep 1277.3 (1425.8) -> 1277.3 (1441.8) MB, 1663.5 / 0.4 ms [allocation failure] [GC in old space requested]. 2019-10-09T08:38:06.4714473Z 38031 ms: Mark-sweep 1277.3 (1441.8) -> 1277.3 (1441.8) MB, 1692.1 / 0.4 ms [allocation failure] [GC in old space requested]. 2019-10-09T08:38:06.4714924Z 39754 ms: Mark-sweep 1277.3 (1441.8) -> 1281.5 (1425.8) MB, 1722.7 / 0.4 ms [last resort gc]. 2019-10-09T08:38:06.4715295Z 41470 ms: Mark-sweep 1281.5 (1425.8) -> 1285.6 (1425.8) MB, 1715.7 / 0.4 ms [last resort gc]. 2019-10-09T08:38:06.4715410Z 2019-10-09T08:38:06.4715468Z 2019-10-09T08:38:06.4715697Z <--- JS stacktrace ---> 2019-10-09T08:38:06.4715737Z 2019-10-09T08:38:06.4715815Z ==== JS stack trace ========================================= 2019-10-09T08:38:06.4715869Z 2019-10-09T08:38:06.4716274Z Security context: 0x3408681cfb39 <JS Object> 2019-10-09T08:38:06.4716784Z 1: defer(aka defer) [/home/vsts/work/_tasks/AzureRmWebAppDeployment_497d490f-eea7-4f2b-ab94-48d9c1acdcb1/4.157.1/node_modules/q/q.js:628] [pc=0x971c62561f8] (this=0x340868104381 <undefined>) 2019-10-09T08:38:06.4717280Z 2: then [/home/vsts/work/_tasks/AzureRmWebAppDeployment_497d490f-eea7-4f2b-ab94-48d9c1acdcb1/4.157.1/node_modules/q/q.js:~826] [pc=0x971c63134fc] (this=0x117c14377d29 <an Object with map 0x3040d32ff... 2019-10-09T08:38:06.4717388Z 2019-10-09T08:38:06.9485221Z 1: node::Abort() [/home/vsts/agents/2.155.1/externals/node/bin/node] 2019-10-09T08:38:06.9490805Z 2: 0x109b9cc [/home/vsts/agents/2.155.1/externals/node/bin/node] 2019-10-09T08:38:06.9495467Z 3: v8::Utils::ReportApiFailure(char const*, char const*) [/home/vsts/agents/2.155.1/externals/node/bin/node] 2019-10-09T08:38:06.9499614Z 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/home/vsts/agents/2.155.1/externals/node/bin/node] 2019-10-09T08:38:06.9505452Z 5: v8::internal::Handle<v8::internal::JSFunction> v8::internal::Factory::New<v8::internal::JSFunction>(v8::internal::Handle<v8::internal::Map>, v8::internal::AllocationSpace) [/home/vsts/agents/2.155.1/externals/node/bin/node] 2019-10-09T08:38:06.9511055Z 6: v8::internal::Factory::NewFunction(v8::internal::Handle<v8::internal::Map>, v8::internal::Handle<v8::internal::SharedFunctionInfo>, v8::internal::Handle<v8::internal::Context>, v8::internal::PretenureFlag) [/home/vsts/agents/2.155.1/externals/node/bin/node] 2019-10-09T08:38:06.9516875Z 7: v8::internal::Factory::NewFunctionFromSharedFunctionInfo(v8::internal::Handle<v8::internal::SharedFunctionInfo>, v8::internal::Handle<v8::internal::Context>, v8::internal::PretenureFlag) [/home/vsts/agents/2.155.1/externals/node/bin/node] 2019-10-09T08:38:06.9523692Z 8: v8::internal::Runtime_NewClosure_Tenured(int, v8::internal::Object**, v8::internal::Isolate*) [/home/vsts/agents/2.155.1/externals/node/bin/node] 2019-10-09T08:38:06.9525056Z 9: 0x971c5d092a7

[LeoSong121]

I am facing same issue.

[MrzJkl]

Solution for Azure Pipeline in Azure DevOps:

- task: PowerShell@2
  displayName: Build
  env:
    NODE_OPTIONS: --max_old_space_size=16384

Set the Environment Variable NODE_OPTIONS with value --max_old_space_size=16384 in your Build-Task.

[appasaheb4]

i am facing same issue. any solution?

[appasaheb4]

i am facing same issue. any solution?

yes my issue fixed. don't use @matrial-ui/icons use alternative lib for icons

[github-actions]

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days