Deploy VAST to AWS Lambda

[rdettai]

This changes adds the possibility to deploy VAST on AWS Lambda and Fargate. This is a more interesting feature set as it will enable connecting to the VAST server on Fargate through client VAST instances on Lambda

📝 Checklist

• All user-facing changes have changelog entries.
• The changes are reflected on docs.tenzir.com/vast, if necessary.
• The PR description contains instructions for the reviewer, if necessary.

🎯 Review Instructions

This PR can be broken down into:

• the terraform scripts to deploy fargate task definition and the lambda function. It will create a subnet in a provided vpc where all this tooling will execute
• a wrapper image around VAST for the lambda function, containing the VAST CLI and the AWS Lambda runtime interface (Dockerfile + python handler)
• a makefile to simplify the interaction with the deployment script and test the deployment

[dispanser]

Great work, thank you so much.

I've tested it following the readme, and was able to deploy vast, import data, query data, ... so everything works as expected.

Two points:

1. I'm not sure the approach using environment variables / default.env is optimal. In my previous usages of terraform, I always relied on tfvars files to inject behavior (and some defaults for the variables). I can see how the approach taken here makes working with the Makefile easier, but it stops us from using terraform directly.
2. Now that you added the ability to directly execute commands on the ecs task using make execute-command, the existence of lambda functions is not strictly necessary any more. It's a nice showcase for how to integrate, and lambda is practically free, so it's not really a problem.

My suggestion would be to move forward as is, and see how this actually gets used and adapt it from these learnings.

[rdettai]

@dispanser

I'm not sure the approach using environment variables / default.env is optimal. In my previous usages of terraform, I always relied on tfvars files to inject behavior (and some defaults for the variables). I can see how the approach taken here makes working with the Makefile easier, but it stops us from using terraform directly.

I don't love this make import either because it is kind of a hack, but it works well in practice. You can still use tfvars to provide the variables if you want to interact with terraform directly: terraform apply -var-file="yourfile.tfvars"

Now that you added the ability to directly execute commands on the ecs task using make execute-command, the existence of lambda functions is not strictly necessary any more. It's a nice showcase for how to integrate, and lambda is practically free, so it's not really a problem.

Yeah, I think I over-engineered this a bit 😅. But I believe it can be very useful if you want to add extra tooling around vast import while keeping the server image to its bare minimum. Ideally, the vast server should be able to run on a much slimmer image (maybe scratch?), and in that case, going through lambda would be a very elegant way to keep the linux flavor you need/like to use the client CLI.

[dominiklohmann]

Thanks for incorporating the CI changes as well.

[dominiklohmann]

@rdettai we require all commits to have a valid signature, which is why your PR cannot be merged currently. Do you have the setup for signing commits locally so you can rebase, or should I rebase this PR leaving you as the author of the commits and me as the (signing) committer?

[rdettai]

I'll setup the signature and rebase

[rdettai]

@dominiklohmann done! you can re-activate auto-merge ;-)