Logs should live in /var/vcap/sys/log, not /var/vcap/store

[bgandon]

Hi,

I recently upgraded from release v6.3.6 to v6.4.0, and I'm facing 3 issues.

1. On the shield/0 VM, postgres startup produces 1.7 GB of logs which is quite a lot.

# du -sh /var/vcap/store/postgres-9.4/pg_log/startup.log
1.7G	/var/vcap/store/postgres-9.4/pg_log/startup.log

By the way, shouldn't these startup logs be located inside the /var/vcap/data/sys/log/postgres directory?

2. On the shield/0 VM, the postgres job produces 27+ GB of logs each day. This fills my production disk very quickly.

For example, I recreated the shield/0 VM yesterday and here is what I have:

# du -sh /var/vcap/data/sys/log/postgres/postgresql.log
27G	/var/vcap/data/sys/log/postgres/postgresql.log

I didn't find a way to control the verbosity of these logs in the postgres-9.4 job spec. Is there any other way?
Would it be possible for the BOSH release to implement some rolling policies on these log files? Or is it up to the deployer to implement some BOSH add-on that does the trick?

3. On the shield/0 VM, the /var/vcap/jobs/shield-agent/bin/ctl startup script has an issue producing this error:

# tail  /var/vcap/data/sys/log/monit/shield-agent.log
/var/vcap/jobs/shield-agent/bin/ctl: line 22: syntax error near unexpected token `}'

This is due to both if_p("shield.agent.autoprovision") and if_p("shield.agent.daemon_public_key") evaluating to false and not rendering their respective blocks. Even when these are set in the deployment properties, which is strange. Anyway, this issue could easily be circumvented with a mere call to /bin/true within the curly braces so that the block is never empty from a Bash syntax point of view.

[jhunt]

Interesting. In no case should shield.agent.autoprovision AND shield.agent.daemon_public_key be omitted. Perhaps we should raise an error if we don't get either of them, rather than generate a configuration of an agent that literally cannot be used.

[jhunt]

As for the log level, we can move the logs into /var/vcap/sys/log, but first I'd be curious what types of messages are making it into your logs to contribute to the 27G/day. Can you characterize these messages, and possible supply some samples?

[jhunt]

May also want to split this into separate Github Issues.

[bgandon]

Ok I'll split the issue and answer there

[bgandon]

As for the 3rd issue, it seems that the /var/vcap/store/postgres-9.4/pg_log/startup.log file is full of such statements:

could not write to log file: No space left on device

And the /var/vcap/store 2GB device was filled up by this log file.
This might be related to issue #50 with the repeated error in postgres logs.

All in all, we would really need to put this log file away from persistent storage.

[jhunt]

The main postgresql logs are on /var/vcap/sys/log; the startup log seems to live in the data directory, until postgresql gets configured with the new log_directory.

[bgandon]

Ok, I think we can close this issue and continue the discussion in #50.