Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Start service on boot #42

Closed
ScottG489 opened this issue Feb 4, 2022 · 4 comments
Closed

Start service on boot #42

ScottG489 opened this issue Feb 4, 2022 · 4 comments
Labels
build Build related enhancement New feature or request

Comments

@ScottG489
Copy link
Owner

In the deployed instance, set up conjob to start on boot.

This will help with being able to restart the server unattended knowing the service will come back up automatically.

Once this is done, create a follow up ticket to run the alt service as a persistent rather than one-time spot_type. Then also consider reducing the spot_price substantially since it isn't super important that it's always available since the conjob build doesn't run very often.

@ScottG489 ScottG489 added build Build related enhancement New feature or request labels Feb 4, 2022
@ScottG489
Copy link
Owner Author

For purposes of a spot instance, the solution to this would be a little more complicated.

When a spot instance is terminated, it's the same as terminating a typical instance. This means that all the setup (mainly via ansible) won't be done. So it won't be as simple as just starting the service again.

However, I think this would still be valuable for just being able to restart an instance normally. We'll likely want to create a oneshot systemd service that will run the conjob container when the VM is starting. This should work fine as long as the container was not removed.

It should also work well for spot instances as long as they aren't terminated. I'm doing work now to have spot instances stopped rather than terminated. They should start back up again automatically when there is spot instance availability again.

@ScottG489
Copy link
Owner Author

ScottG489 commented Mar 3, 2022

I provisioned the alt server with a spot instance specifying to stop the instance on interruption. The instance started back up and the container was still there.

Creating a oneshot service to start the container should work fine.

However, there's a problem that needs to be resolved first. When the instance starts back up, it seems like it's possible for it to have a new IP address. This happened, and DNS was no longer mapping to the correct IP. I see two ways this could be fixed:

  • Allocate a static IP on creation
  • Have the DNS record use an alias to point to the instance instead of an IP

For the former, I think I'd want to use an Elastic IP. It seems that there's no charge for having a single IP associated with a EC2 instance. There's a small charge for when it's associated and the machine is down. This applies to us since the alt server will be stopped occasionally since it's a spot instance. However, I don't believe it's stopped for very long and I doubt this will be a problem. Still something to potentially keep an eye on.

For the latter, I'm not entire sure what this would look like, but I'm just aware of alias records and how they can point to a specific AWS resource.

I'm not sure which solution I would prefer. I may need to look into each to see what the implementation would look like. The elastic IP solution seems somewhat clean, although it's another piece to deal with and it could come with other complications or limits and I feel like I shouldn't have to rely on a specific IP and the DNS is really what should be handling this automatically. The alias solution has a hint of hackyness to it since aliases are an AWS specific thing.


After a little research it appears that you can't create an alias directly to an EC2 instance, so the Elastic IP will hopefully be a clean implementation.

@ScottG489
Copy link
Owner Author

So far the EIP solution was implemented and it seems like the instance restarts just fine.

Next up is to implement a service that runs to start conjob on boot. I think a "oneshot" will work.


NOTE: most of the work done so far has just been for the alt-env and needs to be carried over to the main service.

@ScottG489
Copy link
Owner Author

Fixed by 47c48f6.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build Build related enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant