Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resolve blobstore logging warning [#174649741] #2279

Merged
merged 2 commits into from
Mar 25, 2021
Merged

Conversation

rkoster
Copy link
Contributor

@rkoster rkoster commented Sep 18, 2020

What is this change about?

The blobstore log contained a warning which has confused people while debugging other blobstore issues.
The log line:

nginx: [alert] could not open error log file: open() "/var/vcap/packages/nginx/logs/error.log" failed (30: Read-only file system)

This was caused by the fact that the default prefix /var/vcap/packages/nginx was not writable.
By changing the prefix (using the -p flag) to /var/vcap/data/blobstore (which is writable), this issue was resolved.

This change also increases the log level to include warning.
Which resulted in the following log line:

 [warn] 6#0: 8192 worker_connections exceed open file resource limit: 1024

This was fixed by increasing the open_files limit via bpm to 8192.

While working on this I also noticed nginx was running in daemon mode (which makes sense under bpm).
However it still had a pid location configured which seemed unnecessary.

Please provide contextual information.

https://www.pivotaltracker.com/story/show/174649741

What tests have you run against this PR?

Manual testing to verify warnings are ending up in the correct file:

==> /var/vcap/sys/log/blobstore/error.log <==
2020/09/18 10:08:33 [warn] 6#0: 8192 worker_connections exceed open file resource limit: 1024

This warning should not be displayed anymore after this PR is merged since the limit in bpm is increased.
Lowering the limit manually in the bpm config and restarting the blobstore should work.

How should this change be described in bosh release notes?

Improved blobstore error logging

Does this PR introduce a breaking change?

No

@cf-gitbot
Copy link

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/174879650

The labels on this github issue will be updated when the story is started.

cunnie pushed a commit that referenced this pull request Mar 25, 2021
We bump nginx because the versions of nginx 1.19.5+ include a flag
(`-e`) that eliminates a harmless-but-concerning log message:

```
nginx: [alert] could not open error log file: open() "/var/vcap/packages/nginx/logs/error.log" failed (30: Read-only file system)
```

[#176851693]
[#2279]

Signed-off-by: Brian Cunnie <bcunnie@vmware.com>
cunnie pushed a commit that referenced this pull request Mar 25, 2021
- Place `.pid` file in bpm-friendly directory
- Use the `-e` flag (for error file) in lieu of `-p`; it reduces the
need for the following gyrations to make things work:
  - Don't create directories or links as part of `pre-start`; they are no
    longer necessary.
  - Don't add additional volumes permissions; we don't use them

[#176851693]
[#2279]

Signed-off-by: Brian Cunnie <bcunnie@vmware.com>
@cunnie cunnie merged commit d46f4b0 into master Mar 25, 2021
@cunnie
Copy link
Member

cunnie commented Mar 25, 2021

@rkoster Thanks for the excellent work!

@mrosecrance mrosecrance deleted the fix-blobstore-log branch April 19, 2021 22:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants