Add config parameter to get fog debug loggings

[WeiQuan0605]

Thanks for contributing to the capi_release. To speed up the process of reviewing your pull request please provide us with:

• A short explanation of the proposed change:
  We'd like to add environment variable to get more fog debug loggings when cc.logging_level is set to debug.
  We opted for a reuse of the variable to keep the config of cc simple.

  The documented behaviour of fog is that debug logging is activated when setting the DEBUG env variable of the process, see http://fog.io/about/getting_started.html.

• An explanation of the use cases your change solves
  After adding the environment variable, we can get more fog debug loggings.

• Links to any other associated PRs

• I have viewed signed and have submitted the Contributor License Agreement

• I have made this pull request to the develop branch

• I have run CF Acceptance Tests on bosh lite
  No, I just tested it on several sap landscapes.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/174316536

The labels on this github issue will be updated when the story is started.

[cwlbraa]

Hi! you should be able to set these environment variables pretty easily on the BOSH VMs in most debug cases...

Do you need them to persist through deploys?

[WeiQuan0605]

Hi,
I have tried to set the environment variable directly on the BOSH VMs, i.e. added "export DEBUG=true" into the file /var/vcap/jobs/cloud_controller_ng/bin/cloud_controller_ng, then I restarted the cloud_controller_ng. But unfortunately it didn't work. I checked the processes on BOSH VMs, the added Environment Variable didn't exist, and our VMs were also broken, which I needed to recreate the VMs.
After reading the Doc: https://bosh.io/docs/bpm/transitioning/#environment-variables, We decided to add the environment variable this way(as in PR).
We have tested on our several landscapes, If we use this way to add environment variable, we can get fog debug loggings on VMs and also in our infrastructure ELK.

[stephanme]

We observe sporadic blocked CC operations related to blobstore. Recent problems happened on worker jobs on api VMs during upload of blobs to Azure blobstore. Enabling fog debug logging would provide a bit more context to understand the issue better.

Sporadic = once a week or so on a well used CF foundation (medium sized = 12 api VMs and ~250 cells).
When the issue happens, pretty soon all worker jobs on all api VMs are dead which leads to a major cf-push outage.

With permanently enabled debug logging, we can collect statistics and try to correlate it with the IaaS blobstore logs. Maybe this helps. The workers hang when trying to write to a socket. Might be related to network problems, badly configured timeouts etc.

[Gerg]

Hey @WeiQuan0605 @stephanme,

I'm curious if y'all have tried following the steps described in this document: https://community.pivotal.io/s/article/how-to-enable-request-logging-from-cloud-controller-to-external-blobstore?language=en_US

[stephanme]

Hi @Gerg,

thanks for pointing to this document. @WeiQuan0605 extended the PR so that fog/blobstore logging works on all IaaS including AliCloud (AliCloud requires additional configuration for log path in cc.*.fog_connection).

Enabling fog logging manually doesn't really help because regular automated CF updates via bosh (biweekly updates + hotfixes possible at any time) would switch it off again. And as mentioned, we need to hunt a very sporadic problem on Azure.

Would it help when we de-couple the fog logging configuration from cc.logging_level, e.g. by introducing a new parameter cc.fog_logging_level? This way we would not change existing CC behavior.

Best regards,
Stephan

[Gerg]

I think it makes sense to break this out into its own property, since operators may want to toggle it independently of the general CC log level.

I would argue that this property should be a boolean toggle instead of a log level. The current check on whether the log level is exactly "debug" won't work for other log levels above debug: https://github.com/cloudfoundry/steno#log-levels. Also, I don't think the fog debug logs offer any gradation by log level.

I propose something similar to the existing cc.log_db_queries property. Maybe something like cc.log_fog_requests (open to other names here)?

[WeiQuan0605]

Hi @Gerg,

thanks for the advice. As you recommended, I added a new boolean property cc.logging_fog_requests to enable the fog debug loggings in my PR. Please let us know if these look good. Thanks!

Best regards,
Wei

[Gerg]

I merged this manually in 8fc0101 and c6af5e4. This should be included in the next capi-release.

Note that I renamed the configuration property to cc.log_fog_requests to match the cc.db_logging_level naming convention.

Thank you for contributing to capi-release!