Add functional testing

[xlr-8]

Issue type

• Feature Idea

Summary

The project has only unit tests, having functional testing would be a great improvement.

Details

There are now possibilities to test Lambda code locally, such as: https://github.com/awslabs/aws-sam-local, this seems also possible on the Lambda console page. However the SAM project doesn't have go support quite yet sadly according to their github page.

It would be nice to investigate, as well as keep on eye on it, or find similar open-source projects to have functional testing, even if events or mocked, at least test the code from end to end.

That would make it easier, save cost in infrastructure creation to actually test it properly and also make contribution/refactoring safer to do.

[lenucksi]

As of current, the aws-sam-local project is marked as supporting Go.
Looks like it started being supported with version 0.2.6 released in late January 2018.

[cristim]

The same autospotting binary that runs in Lambda can be executed locally and the other way round, and we can pass the env vars as CLI flags, so from this perspective we're good for local execution. SAM local is pretty much all about the event data and API Gateway, which we don't use for now.

I think our biggest challenge is emulating EC2 resources so that we don't need to actually run them and incur costs, much like the moto library does for Python. Not sure what's the best way to emulate those locally in a way that's not causing more problems due to inconsistency with the real thing.

AutoSpotting is cloud-native which means it pretty much requires cloud infrastructure to run against and you can't really run it locally.

[lenucksi]

I've just looked around a bit and found the following things:

• http://docs.getmoto.org/en/latest/docs/server_mode.html Moto has a mode presenting it's mocked endpoints via HTTP. This combined with https://docs.aws.amazon.com/sdk-for-go/api/aws/endpoints/ might be closer to what we are looking for.
• We'd probably need some bit of Python scripting preparing an expected region full of EC2 instances in ASG's and then run Autospotting against that with the expectations from the Python. Writing them down as some sort of JSON and loading from / checking against that from both languages is probably a somewhat acceptably horrible way to do so. Also, maybe handling the Python in the setup phase of Go tests might be an option.
• However, it does look like there might be parts of the Autoscaling API missing in moto: https://github.com/spulec/moto/blob/master/IMPLEMENTATION_COVERAGE.md -> terminate_instance_in_auto_scaling_group is missing. We do appear to have detach&terminate as an option, though.

[cristim]

That's exactly what I tried to warn against, that the moto mocked resources have different behavior than the real AWS resources they are supposed to mimic, and then we spend time troubleshooting the differences.

But I am open to try it out and see how it goes, we should just keep the option to revert if we then repeatedly run into issues with the moto mocking

[cristim]

We now have support for running the code locally passing an event file, and that can be used to simulate some behavior, but this still needs to be orchestrated carefully due to the event driven nature of the current implementation.

I think integration tests would be nice to have though, but they're not a priority for me at the moment