Fetching contributors…
Cannot retrieve contributors at this time
320 lines (222 sloc) 16.2 KB
title: Troubleshooting Application Deployment and Health
- Routing
- Release Integration
<strong><%= modified_date %></strong>
<% $this_topic = 'troubleshoot-app-health' %>
Refer to this topic for help diagnosing and resolving common issues when you deploy and run applications on Cloud Foundry.
## <a id='scenarios'></a>Common Issues ##
The following sections describe common issues you might encounter when attempting to deploy and run your application, and possible resolutions.
### <a id='time'></a>cf push Times Out ###
If your deployment times out during the upload or staging phase, you may receive one of the following error messages:
* `504 Gateway Timeout`
* `Error uploading application`
* `Timed out waiting for async job JOB-NAME to finish`
If this happens, do the following:
* **Check your network speed.**
Depending on the size of your application, your `cf push` could be timing out because the upload is taking too long.
We recommended an Internet connection speed of at least 768 KB/s (6 Mb/s) for uploads.
* **Make sure you are pushing only needed files.**
By default, `cf push` will push all the contents of the current working directory.
Make sure you are pushing only the directory for your application.
If your application is too large, or if it has many small files, Cloud Foundry may time out during the upload.
To reduce the number of files you are pushing, ensure that you push only the directory for your application, and remove unneeded files or use the `.cfignore` file to [specify excluded files](prepare-to-deploy.html#exclude).
* **Set the CF\_STAGING\_TIMEOUT and CF\_STARTUP\_TIMEOUT environment variables.**
By default your app has 15 minutes to stage and 5 minutes to start.
You can increase these times by setting `CF_STAGING_TIMEOUT` and `CF_STARTUP_TIMEOUT`.
Type `cf push -h` at the command line for more information.
* **If your app contains a large number of files, try pushing the app repeatedly.**
Each push uploads a few more files.
Eventually, all files have uploaded and the push succeeds.
This is less likely to work if your app has many small files.
### <a id='upload'></a>App Too Large ###
If your application is too large, you may receive one of the following error messages on `cf push`:
* `413 Request Entity Too Large`
* `You have exceeded your organization's memory limit`
If this happens, do the following:
* **Make sure your org has enough memory for all instances of your app.**
You will not be able to use more memory than is allocated for your
To view the memory quota for your org, use `cf org ORG_NAME`.
Your total memory usage is the sum of the memory used by all applications in all spaces within the org.
Each application's memory usage is the memory allocated to it
multiplied by the number of instances.
To view the memory usage of all the apps in a space, use `cf apps`.
* **Make sure your application is less than 1 GB.**
By default, Cloud Foundry deploys all the contents of the current working directory.
To reduce the number of files you are pushing, ensure that you push only the directory for your application, and remove unneeded files or use the `.cfignore` file to [specify excluded files](prepare-to-deploy.html#exclude).
The following limits apply:
* The app files to push cannot exceed 1 GB.
* The droplet that results from compiling those files cannot exceed 1.5 GB.
Droplets are typically a third larger than the pushed files.
* The combined size of the app files, compiled droplet, and buildpack cache cannot total more than 4 GB of space during staging.
### <a id='detect'></a>Unable to Detect a Supported Application Type ###
If Cloud Foundry cannot [identify an appropriate buildpack](../../buildpacks/detection.html) for your app, you will see an error message that states `Unable to detect a supported application type`.
You can view what buildpacks are available with the `cf buildpacks` command.
If you see a buildpack that you believe should support your app, refer to the [buildpack documentation](../../buildpacks/) for details about how that buildpack detects applications it supports.
If you do not see a buildpack for your app, you may still be able to push your application with a [custom buildpack](../../buildpacks/custom.html) using `cf push -b` with a path to your buildpack.
### <a id='deploy_fail'></a> App Deploy Fails ###
Even when the deploy fails, the app might exist on <%=vars.product_short%>.
Run `cf apps` to review the apps in the currently targeted org and space. You might be able to correct the issue using the <%=vars.dev_console_2%>, or you might have to delete the app and redeploy.
Common reasons deploying an app fails include the following:
* You did not successfully create and bind a needed service instance to the app, such as a PostgreSQL or MongoDB service instance. Refer to Step 3: Create and Bind a Service Instance for a RoR Application.
* You did not successfully create a unique URL for the app. Refer to the troubleshooting tip App Requires Unique URL.
### <a id='unique_url'></a> App Requires Unique URL ###
<%=vars.product_short%> requires that each app that you deploy has a unique URL. Otherwise, the new app URL collides with an existing app URL and <%=vars.product_short%> cannot successfully deploy the app. You can resolve this issue by running `cf push` with either of the following flags to create a unique URL:
* `-n` to assign a different HOST name for the app.
* `--random-route` to create a URL that includes the app name and random words. Using this option might create a long URL, depending on the number of words that the app name includes.
### <a id='start'></a>App Fails to Start ###
After `cf push` stages the app and uploads the droplet, the app may fail to start, commonly with a pattern of starting and crashing similar to the following example:
<pre class="terminal">
-----> Uploading droplet (23M)
0 of 1 instances running, 1 starting
0 of 1 instances running, 1 down
0 of 1 instances running, 1 failing
Start unsuccessful
If this happens, try the following:
* **Find the reason app is failing and modify your code.**
Run `cf events APP-NAME` and `cf logs APP-NAME --recent` and look for
messages similar to this:
<pre class="terminal">
2018-07-20T15:53:26.00-0700 app.crash app-name index: 2, reason: CRASHED, cell\_id: d874ad05-d0ca-4c63-9f5a-0b1ddd90dd5d, instance: f406c53e-b1ca-4a0f-6140-dddd, exit\_description: APP/PROC/WEB: Exited with status 1
These messages may identify a memory or port issue.
If they do, take that as a starting point when you re-examine and fix your
application code.
* **Make sure your application code uses the `PORT` environment variable.**
Your application may be failing because it is listening on the wrong port.
Instead of hard coding the port on which your application listens, use the `PORT` environment variable.
For example, this Ruby snippet assigns the port value to the `listen_here`
`listen_here = ENV['PORT']`
For more examples specific to your application framework, see the
appropriate [buildpacks documentation](../../buildpacks/) for your app's language.
* **Make sure your app adheres to the principles of the
[Twelve-Factor App]( and [Prepare to Deploy an Application]
These texts explain how to prevent situations where your app builds locally but
fails to build in the cloud.
* **Verify the timeout configuration of your app.** Application health checks use a timeout setting when an app starts up for the first time. See [Using Application Health Checks](./healthchecks.html). If an application fails to start up due to health check timeout, you might see messages in the logs similar to the following:
<pre class="terminal">
2017-01-30T14:07:20.39-0800 [CELL/0] OUT Creating container
2017-01-30T14:07:20.65-0800 [CELL/0] OUT Successfully created container
2017-01-30T14:07:22.30-0800 [CELL/0] OUT Starting health monitoring of container
2017-01-30T14:08:23.52-0800 [CELL/0] ERR Timed out after 1m0s:
health check never passed.
2017-01-30T14:08:23.57-0800 [CELL/0] OUT Destroying container
2017-01-30T14:08:23.59-0800 [API/0] OUT Process has crashed with type: "web"
2017-01-30T14:08:23.59-0800 [CELL/0] OUT Creating container
2017-01-30T14:08:23.60-0800 [API/0] OUT App instance exited with guid 91086440-bac0-44f0-808f-a034a1ec2ed0
payload: {"instance"=>"", "index"=>0, "reason"=>"CRASHED",
"exit\_description"=>"2 error(s) occurred:\n\n* 1 error(s)
occurred:\n\n* Exited with status 6\n* 2 error(s)
occurred:\n\n* cancelled\n* cancelled", "crash\_count"=>1,
2017-01-30T14:08:23.83-0800 [CELL/0] OUT Successfully destroyed container
2017-01-30T14:08:23.84-0800 [CELL/0] OUT Successfully created container
2017-01-30T14:08:25.41-0800 [CELL/0] OUT Starting health monitoring of container
### <a id='out-of-memory'></a>App consumes too much memory, then crashes ###
An app that `cf push` has uploaded and started can crash later if it uses
too much memory.
**Make sure your app is not consuming more memory than it should.**
When you ran `cf push` and `cf scale`, that configured a limit on the amount
of memory your app should use.
Check your app's actual memory usage.
If it exceeds the limit, modify the app to use less memory.
### <a id='routing-conflict'></a>Routing Conflict ###
<%= partial 'routing_conflict' %>
If you suspect a routing conflict, run `cf routes` to check the routes in your installation.
If two apps share a route outside of a Blue-Green deploy strategy, choose one app to re-assign to a different route and follow the procedure below:
1. Run `cf unmap-route YOUR-APP-NAME OLD-ROUTE` to remove the existing route from that app.
1. Run `cf map-route YOUR-APP-NAME NEW-ROUTE` to map the app to a new, unique route.
## <a id='info'></a>Gather Diagnostic Information ##
Use the techniques in this section to gather diagnostic information and troubleshoot app deployment issues.
### <a id='env'></a>Examine Environment Variables ###
`cf push` deploys your application to a container on the server.
The environment variables in the container govern your application.
You can set environment variables in a manifest created before you deploy.
See [Deploying with Application Manifests](./manifest.html).
You can also set an environment variable with a `cf set-env` command followed
by a `cf push` command.
You must run `cf push` for the variable to take effect in the container
Use the `cf env` command to view the environment variables that you have set using the `cf set-env` command and the variables in the container environment:
<pre class="terminal">
$ cf env my-app
Getting env variables for app my-app in org My-Org / space development as admin...
"p-mysql-n/a": [
"credentials": {
"label": "p-mysql-n/a",
"name": "p-mysql",
"syslog_drain_url": "",
"tags": ["postgres","postgresql","relational"]
my-env-var: 100
### <a id='logs'></a>View Logs ###
To view app logs streamed in real-time, use the `cf logs APP-NAME` command.
To aggregate your app logs to view log history, bind your app to a syslog drain service. For more information, see [Streaming Application Logs to Log Management Services](./../services/log-management.html).
<p class="note"><strong>Note</strong>: The Diego architecture does not support the <code>cf files</code> command.</p>
### <a id='trace'></a>Trace Cloud Controller REST API Calls ###
If a command fails or produces unexpected results, re-run it with HTTP tracing
enabled to view requests and responses between the Cloud Foundry Command Line Interface (cf CLI) and the Cloud Controller REST API.
For example:
* Re-run `cf push` with `-v`:
`cf push APP-NAME -v`
* Re-run `cf push` while appending API request diagnostics to a log file:
These examples enable HTTP tracing for a single command only.
To enable it for an entire shell session, set the variable first:
`export CF_TRACE=true`
<p class="note"><strong>Note</strong>: <code>CF_TRACE</code> is a local environment variable that modifies the behavior of the cf CLI.
Do not confuse <code>CF_TRACE</code> with the <a href="#env">variables in the container environment</a> where your apps run.</p>
### <a id='zipkin_trace'></a>Analyze Zipkin Trace IDs
When the Zipkin feature is enabled in Cloud Foundry, the Gorouter adds or forwards Zipkin trace IDs and span IDs to HTTP headers. For more information about what the Gorouter provides in the HTTP header, see the [HTTP Headers](../../concepts/http-routing.html#http-headers) section of the _HTTP Routing_ topic.
After adding Zipkin HTTP headers to app logs, developers can use `cf logs myapp` to correlate the trace and span ids logged by the Gorouter with the trace ids logged by their app. To correlate trace ids for a request through multiple apps, each app must forward appropriate values for the headers with requests to other applications.
## <a id='cf-commands'></a>Use Troubleshooting Commands ##
You can investigate app deployment and health using the cf CLI.
Some cf CLI commands may return connection credentials.
Remove credentials and other sensitive information from command output before you post the output a public forum.
* `cf apps`: Returns a list of the applications deployed to the current space
with deployment options, including the name, current state, number of instances,
memory and disk allocations, and URLs of each application.
* `cf app APP-NAME`: Returns the health and status of each instance of a specific application in the current space, including instance ID number, current
state, how long it has been running, and how much CPU, memory, and disk it is using.
<p class="note"><strong>Note</strong>: CPU values returned by <code>cf app</code> show the total usage of each app instance on all CPU cores on a host VM, where each core contributes 100%. For example, the CPU of a single-threaded app instance on a Diego cell with one core cannot exceed 100%, and four instances sharing the cell cannot exceed an average CPU of 25%. A multi-threaded app instance running alone on a cell with eight cores can draw up to 800% CPU.</p>
* `cf env APP-NAME`: Returns environment variables set using `cf set-env` and variables existing in the container environment.
* `cf events APP-NAME`: Returns information about application crashes, including error codes. Shows that an app instance exited. For more detail, look in the application logs. See []( for a list of Cloud Foundry errors.
* `cf logs APP-NAME --recent`: Dumps recent logs.
See [Viewing Logs in the Command Line Interface](./streaming-logs.html#view).
* `cf logs APP-NAME`: Returns a real-time stream of the application STDOUT and
STDERR. Use **Ctrl-C** (^C) to exit the real-time stream.
* `cf files APP-NAME`: Lists the files in an application directory.
Given a path to a file, outputs the contents of that file. Given a path to a
subdirectory, lists the files within. Use this to [explore](#logs) individual
<p class="note"><strong>Note</strong>: Your application should direct its logs to STDOUT and STDERR.
The <code>cf logs</code> command also returns messages from any <a href="">log4j</a> facility that you configure to send logs to STDOUT.</p>
## <a id='ssh'></a>Access Apps with SSH ##
If you need to troubleshoot an instance of an app, you can gain SSH access to the app with the SSH proxy and daemon. See the [Application SSH Overview](./app-ssh-overview.html) topic for more information.
## <a id='request-to-instance'></a>Send Requests to App Instance ##
To obtain debug data without SSH, you can make HTTP requests to a specific instance of an app by using the `X-CF-APP-INSTANCE` HTTP header. See the [App Instance Routing](../../concepts/http-routing.html#app-instance-routing) section of the <em>HTTP Routing</em> topic for more information.