Provision new server for HAProxy w/ new configuration

[GTChase]

User Story:
As a developer
I want to a provision a new instance to install the new HAProxy configurations
So that we dont disrupt the production services

Acceptance Criteria
GIVEN I am a developer
WHEN I deploy the new HAproxy services
THEN it should be deployed on a new instance
AND that instance is separate from the current production instance/production shouldn't be interrupted

[gdixon]

HAProxy update:

• Fixed up the HAProxy configuration for a bare-metal installation
• Stood up a new instance and installed all requirements manually
• Found 100s of bugs in the config and spent the night working through them, I feel like I could spot a fire hydrant with my eyes closed now!

Bugs include:

• We need to use hdr('x-forwarded-for') rather than src for IP bucketing and this has terrible support in Lua so we had to rewrite the relevant parts to carry through the IP from HAProxy
• DNS lookups are failing - I've put a workaround in place - but this should be fixed as we've introduced a SPoF
• Verifications were intermittently failing due to the way in which we were catching a submission - we're now directing all POSTs when in 'needs verification' mode to the validation script
• ENV variables are not available to the HAProxy user even though they are being sourced globally so they've now been baked in to templates using sed - the only sensitive piece of data here is the google-recaptcha keys - but even these can be generated by anyone (the safety mechanism here is that google only validates if the req comes from our domain) - so no real secrets, the .env is purely config!
• and others...

A working POC can be found here: https://hap-test.support.gitcoin.co/

Last steps:

• ✅ Switch over to gitcoin.co recaptcha-v2 keys
• ✅ Point the ALB at the new instance
• ✅ Install datadog and apply the appropriate config
• ✅ Push changes to haproxy to github
• ✅ Push changes to gdeploy to expose the new instance to the team