Cloud Foundry running in Bosh Lite on AWS intermittently returns 502 when making requests to hosted apps

[stupakov]

The Apps Manager team has a bosh-lite running on AWS (http://apps.bosh-lite.appsman.cf-app.com/) that has an intermittent problem: hitting app URLs sometimes returns a 502.

The CAPI team has seen the same problem on their bosh-lite environment (which has been deleted).

@michaelxupiv from CAPI said:

we suspected it was a garden container issue because we tracked the traffic reaching the Diego VM, but somehow that packet was lost when trying to reach the container. It's interesting that we don't see this issue on local bosh-lite.

Here are the logs they captured on their environment before it was torn down.
502app-logs.txt

If you wish to investigate this issue, the Apps Manager team can provide credentials to the bosh lite environment.

This is the runtime that we have deployed there:

+-----------------+------------------------------------------------+-----------------------------------------------------+--------------+
| Name            | Release(s)                                     | Stemcell(s)                                         | Cloud Config |
+-----------------+------------------------------------------------+-----------------------------------------------------+--------------+
| cf-warden       | cf/250                                         | bosh-warden-boshlite-ubuntu-trusty-go_agent/3312.15 | none         |
+-----------------+------------------------------------------------+-----------------------------------------------------+--------------+
| cf-warden-diego | cf/250                                         | bosh-warden-boshlite-ubuntu-trusty-go_agent/3312.15 | none         |
|                 | cflinuxfs2-rootfs/1.45.0                       |                                                     |              |
|                 | diego/1.5.3                                    |                                                     |              |
|                 | garden-runc/1.1.1                              |                                                     |              |

[garden-gnome]

Hi there!

We use Pivotal Tracker to provide visibility into what our team is working on. A story for this issue has been automatically created.

The current status is as follows:

• #138522729 Cloud Foundry running in Bosh Lite on AWS intermittently returns 502 when making requests to hosted apps

This comment, as well as the labels on the issue, will be automatically updated as the status in Tracker changes.

[julz]

Hi @stupakov - bosh-lite still uses garden-linux which is end-of-lifed, so it's a little hard for us to support. I believe you should be able to create a bosh-lite based on garden-runc using the manifest here, I'd be super super interested in knowing if the same behaviour happens with this.

[williammartin]

@julz I recall @stupakov saying when they ran this using local bosh-lite vagrant box it did not occur, only on AWS

[stupakov]

@williammartin - I was talking to @utako and @ljfranklin on Friday and they mentioned that they just observed the problem on local bosh-lite in Vagrant as well.

@julz - sounds like if bosh-lite is to continue being supported, the default configuration it ships with should be updated to use garden-runc as in your link. I'll see what the bosh folks think about this.

[geofffranks]

I ran into a similar problem documented in cloudfoundry/guardian#68 when running on Azure and vSphere. This was using garden-linux on bosh-lite. When using garden-runc at the bosh-lite level, the problem did not occur (with the same CF + bosh-lite manifests). In all cases, diego was using garden-runc

[julz]

Closing since this is really an issue for bosh-lite, we no longer support garden-linux.