AWS dies after a day; three things to fix

[cgoettel]

The current AWS instance is slow as rocks and fills up in about a day. Three things to get this fixed:

• System requirements for OpenCTI say 6 CPU, 16GB RAM, and minimum 32GB disk.
• Log rotation for elasticsearch (maybe others; implemented log rotate for journalctl (see Set hard limit for journald #4) and saw most recently that elasticsearch was filling up the disk)
• Increase disk size to 32GB. This is a separate item because it's not controlled in the instance choice.

[cgoettel]

Disk is EBS, not OS disk. FYI.

[cgoettel]

The code checked into cgoettel-expand-aws is ready to be tested. Notes for when I get to it on Monday:

• Check how the EBS disk is mounted (mounted at /dev/sdf):
  • Do I need to add anything to fstab to get it working?
  • Where are the logs being written? Is that mounted on the new 32GB instance or is it all still going to the OS disk?
• Made some changes to the variable location. Is that working?

[cgoettel]

Update from work and thoughts yesterday:

• The EBS disk is not mounted. It is /dev/nvme1n1. Added code to the script to create a filesystem (if needed) and mount it. It's kinda dirty and will need to be reworked if another disk is added. Alsø did the fstab thing.
• The logs are the real issue on this system. /usr and /var each take up like 40% of the space. Possible solutions (both good and bad (including, but not limited to, awful):
  • Create partitions on the EBS and mount /var and /usr to it. Maybe just /var because that's the one that will grow? Where do the connectors put their stuff? If it's on /usr, that's an issue.
  • Symbolic link /var and /usr to the new mount. Horrific idea. If the mount doesn't come up, the OS is wack.
  • Config changes for each of the applications to put their stuff on the new drive (or symbolic link the config directories, but then we're back to the previous problem). I think this is really the best solution. It sucks and it's a bunch of work (and have to keep these AWS-specific bits separate in the code from the Azure and GCP stuff).
• Variable changes are working.

[cgoettel]

I have failed as an engineer and a researcher. Read one article saying that you can't increase the root volume's size and I took it as gospel. Moron. The extra EBS disk is no longer in the code, root volume increased to 32GB.

[cgoettel]

Changed the VPC and Subnet from being manual to being automated by Terraform. And now I can't access the instance via Systems Manager. Tryna figure out why.

[cgoettel]

I have exhausted myself trying to figure out why #9 is happening. Can't figure it out. Gone back to the previous code and calling this issue quits.