Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Log incomplete requests when stopping CC #2256

Merged
merged 1 commit into from May 18, 2021
Merged

Log incomplete requests when stopping CC #2256

merged 1 commit into from May 18, 2021

Conversation

philippthun
Copy link
Member

@philippthun philippthun commented May 5, 2021

Thanks for contributing to cloud_controller_ng. To speed up the process of reviewing your pull request please provide us with:

  • A short explanation of the proposed change:
    When CC receives the signal to stop execution, it simply aborts outstanding requests. With this change, all incomplete requests will be written into the log file when (after) the Thin server is stopped.

    The request logging code has been restructured similar to Metrics::RequestMetrics.

  • An explanation of the use cases your change solves
    This change enables operators to identify requests that have not been completed due to a restart of Cloud Controller. Without this change it was required to search through the logs and compare the Started ... and Completed ... messages in order to identify which requests were still processed when CC stopped. This helps analyzing potentially blocking and/or slow requests that need to be improved.

  • Links to any other associated PRs

  • I have reviewed the contributing guide

  • I have viewed, signed, and submitted the Contributor License Agreement

  • I have made this pull request to the main branch

  • I have run all the unit tests using bundle exec rake

  • I have run CF Acceptance Tests

When CC receives the signal to stop execution, it simply aborts outstanding requests. Writing request logs has been moved to Logs::RequestLogs (similar to Metrics::RequestMetrics). Logs::RequestLogs provides the possibility to log all incomplete requests; this is called after the thin server has been stopped.
@elenasharma
Copy link
Contributor

When reviewing this, might be handy to check out cloudfoundry/capi-release#195

Copy link
Contributor

@sweinstein22 sweinstein22 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sethboyles and I looked through the changes, agreed that they look reasonable, and ran through the following acceptance steps:

Acceptance:

  • deployed changes to a boshlite
  • set the apps index endpoint to sleep 1000, which is longer than the 30 second graceful shutdown timeout allowed per Invoke restart drain for failed healthcheck capi-release#195
  • ran cf apps -v and saw a call to /v3/apps?order_by=name&space_guids=GUID stall out
  • bosh ssh'd onto the api vm, ran monit stop cloud_controller_ng
  • checked /var/vcap/sys/log/cloud_controller_ng/cloud_controller_ng.log and found instances of log lines mirroring those outlined in this change, Incomplete request: GET \"/v3/apps?order_by=name&space_guids=GUID

image

Thanks for the contribution!

@sweinstein22 sweinstein22 merged commit 3eaf5af into cloudfoundry:main May 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants