feat: make kwil gateway binary available in ec2 instance by VladislavSpassov · Pull Request #141 · trufnetwork/node

VladislavSpassov · 2024-03-20T18:13:44Z

This PR adds a boot up script to ec2 instance to fetch from s3 the kwil gateway binary. It also extracts logic for creating ec2 role into its own function.

Related Issue

Resolves #109.

Motivation and Context

We need to have kwil gateway binary in ec2 instance to be able to execute it.

How Has This Been Tested?

Run manually kwilGatewayBinaryScript. The permission grant needs to be tested on staging or at least I don't know another way currently.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Documentation update (including inline docs)
Other (please describe):

Checklist:

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have added tests to cover my changes.
All new and existing tests passed.

Checklist Explanation:

How to Review this PR:

Additional Information:

…ec2 instance This PR extracts logic for creating ec2 role into its own function. Also it adds a boot up script to ec2 instance to fetch from s3 the kwil gateway binary. Resolves #109.

keep binary out of tmp folder

outerlook · 2024-03-20T19:16:34Z

Hey @VladislavSpassov,
I'll test it soon. However, I'll approve it after #117 gets merged, so we may merge it directly to the new main and keep the gateway feature separate from the 0.7 upgrade, although it's ready for the changes.

VladislavSpassov · 2024-03-20T19:28:56Z

@outerlook sounds great! Keep me in the loop and I will react if there is something to be fixed.

outerlook

Hey, although I saw that permissions were correctly added, I didn't see where you actually load the file from S3 into the instance (if I missed please point it for me)

My suggestion would be using awsec2.InitFile_FromS3Object(...) similar to what we do here:

https://github.com/truflation/tsn-db/blob/c8391f5ab0087678bd82fcf18ef9ee7e4972e781/deployments/infra/cdk_main.go#L99

VladislavSpassov · 2024-03-20T20:54:20Z

Hey, although I saw that permissions were correctly added, I didn't see where you actually load the file from S3 into the instance (if I missed please point it for me)

My suggestion would be using awsec2.InitFile_FromS3Object(...) similar to what we do here:

https://github.com/truflation/tsn-db/blob/c8391f5ab0087678bd82fcf18ef9ee7e4972e781/deployments/infra/cdk_main.go#L99

I am running aws s3 cp s3://kwil-binaries/gateway/kgw_v0.1.2.zip /tmp/kgw_v0.1.2.zip in the script part

change place of newName variable

move binary to /usr/local/bin

outerlook · 2024-03-20T21:29:50Z

I am running aws s3 cp s3://kwil-binaries/gateway/kgw_v0.1.2.zip /tmp/kgw_v0.1.2.zip in the script part

All right! I was still viewing the old PR branch. Now it should be all good to work 👌 I'll test it as the first thing tomorrow

rewrite kwilGatewayBinaryScript

outerlook

Tested and working. kgw binary is accessible by ec2 instance users (if using root user, must access /usr/local/bin/kgw)

markholdex · 2024-03-22T14:47:22Z

@VladislavSpassov @outerlook @MicBun are we missing anything here to merge?

outerlook · 2024-03-22T14:54:39Z

@markholdex

yes, from this context: #141 (comment)

We're waiting for #117 to be merged so we can merge directly to main and not mix up output (or bugs) from gateway work to the 0.7 upgrade.
This should not block any work in progress

VladislavSpassov · 2024-03-25T12:55:46Z

Hey @outerlook, how is the upgrade going, are we merging this? I ask because with the merge my trial period would be done.

outerlook · 2024-03-25T13:08:24Z

Alright, @VladislavSpassov. I'll merge this as it's important for your trial, as for us here it's just a matter of code management.

VladislavSpassov · 2024-03-25T13:15:47Z

Thanks a lot, Raffa!

* refactor: sync repo with kwil v0.7 * merge conflic * merge conflic * resolve conflicts * add .env to ignore * fix whitelist.go * fix typo * fix: setup-scripts (#127) * Delete .idea directory * fix(refactor-0.7): working date-range (#128) * add develop scripts * Refactor getValue and related methods, fixed to query with engine Context parameter was removed to streamline the method calls within the basestream module. The dbid argument was replaced by using scope directly, resulting in more concise and consistent function signatures across the module. * Refactor test suite to use new query execution interface Refactored `basestream_test.go` replacing the context and app setup with a `ProcedureContext` and a mock engine using a new query execution interface. Improved function calls to align with changes and updated test assertions to work with the revised structure and methods. * Add debug build target to Taskfile Implemented a new task `build:debug` for compiling the kwild binary with debug flags. This allows for better debugging experiences by disabling optimizations and inlining. * Update stream IDs in composed.kf Changed stream identifiers to use generic `/stream_b` and `/stream_a` instead of specific external service references. This update simplifies the development process by using placeholder values rather than actual external service streams. * Refactor build and deploy tasks in documentation Consolidated various script calls under single `task` commands for simplicity and readability. Updated the database seed section to reflect changes in order and improved the database query examples for consistency with stream_b. * remove unnecessary transform script * fix(refactor-0.7): working ci and dockerfile (#131) * fix(refactor-0.7): working ci and dockerfile * add docker compose * using setup for ci * hardcoded private key for seeding * revert run acceptance-test on draft * edit readme * put kwil-cli into .build * fix missing .. on database add primitive and hardcode whitelist wallets * add whitelist * add white list wallet * Update .github/workflows/ci.yaml Co-authored-by: Raffael Campos <raffael.rcampos@gmail.com> * replace WHITELIST_WALLETS --------- Co-authored-by: Raffael Campos <raffael.rcampos@gmail.com> * test(refactor-07): enable unit test on ci (#139) feat: enable unit test on ci * fix(refactor-0.7): make CDK infra compatible to latest dockerfile (#136) * Add infrastructure and CDK configuration for Go project Add new CDK configuration files, Go modules, Docker and Git ignore files, and placeholder Go tests for a Go project infrastructure setup. Adjust a docker file reference in `cdk_main.go`. * Remove cdk.iml and update .gitignore Removed the cdk.iml file from version control and added it to .gitignore to prevent it from being tracked. This ensures IDE-specific files do not clutter the repository and maintains a clean workspace for all collaborators. * Add Docker setup for TSN data push Added Dockerfile and updated CI and docker-compose configs to establish the environment for pushing TSN data. The CI workflow is adjusted with Python setup changes and additional steps to verify the compose status. Docker compose configuration now includes persistence and network definitions. * Update .gitignore to ignore all .env files except example The .gitignore file has been modified to ignore any files with the .env prefix. An exception has been added to keep the .env.example file in version control, which is often used for providing configuration templates. * Refactor setup script and isolate wait logic The setup script now allows for an environment variable override for the gRPC URL and extracts the waiting logic for kwild readiness into a separate script. Additionally, the creation of the kwil_cli config file now utilizes the gRPC URL environment variable, improving configurability and adherence to the DRY principle. * Add clear-data task and update README Taskfile now includes a clear-data task to remove local Kwil data and Docker volumes. README instructions have been updated to reflect the new method of clearing data, and non-essential Docker Compose sections have been consolidated for clarity. * Refactor gRPC URL configuration in CI tests * Add vendor directory and go.work.sum to .dockerignore The .dockerignore has been updated to exclude the `vendor/` directory and the `go.work.sum` file. These changes ensure unnecessary directories and files are not included in the Docker build context, optimizing build performance. * Update go-isatty and go-tools to newer versions Upgraded github.com/mattn/go-isatty to v0.0.20 and golang.org/x/tools to v0.16.1. The update includes the removal of older module versions resulting in a cleaner module dependency graph. * Update dependencies in go.mod and go.sum Updated `go-spew` to `v1.1.2-0.20180830191138-d8f796af33cc`, `go-difflib` to `v1.0.1-0.20181226105442-5d4384ee4fb2`, and `golang.org/x/sys` to `v0.16.0`. Also added checksums in `go.sum` for these changes. * Update Docker image deployment logic Removed old build argument logic and streamlined the process of deploying Docker images on an EC2 instance. Added s3 asset for docker-compose file and adjusted service definition to use Docker Compose for service management. * Update instance to use Amazon Linux 2 Changed user data script to support Amazon Linux 2 AMI, ensuring compatibility with AWS environments. The Docker and AWS CLI installation process was adapted for yum package manager and the script now also includes docker-compose setup. Docker permissions are adjusted for the default ec2-user. * Add warning comment on environment variables A warning comment was added to getEnvStringsForService to alert maintainers about the lack of encryption in environment variables within the CloudFormation template and the potential access through service configuration files. The comment recommends switching to encryption if necessary. * Update .gitignore * Update deployments/infra/go.mod Co-authored-by: Michael Buntarman <michaelboentarman@gmail.com> * update import path to reflect module name change * Remove unused test file for CDK main The file `cdk_main_test.go` was removed because it contained only commented-out code and unused tests. This should have no impact on the actual test suite as the file was not in use. --------- Co-authored-by: Michael Buntarman <michaelboentarman@gmail.com> * fix(refactor-0.7): Working CDK (#132) * Add infrastructure and CDK configuration for Go project Add new CDK configuration files, Go modules, Docker and Git ignore files, and placeholder Go tests for a Go project infrastructure setup. Adjust a docker file reference in `cdk_main.go`. * Remove cdk.iml and update .gitignore Removed the cdk.iml file from version control and added it to .gitignore to prevent it from being tracked. This ensures IDE-specific files do not clutter the repository and maintains a clean workspace for all collaborators. * Add Docker setup for TSN data push Added Dockerfile and updated CI and docker-compose configs to establish the environment for pushing TSN data. The CI workflow is adjusted with Python setup changes and additional steps to verify the compose status. Docker compose configuration now includes persistence and network definitions. * Update .gitignore to ignore all .env files except example The .gitignore file has been modified to ignore any files with the .env prefix. An exception has been added to keep the .env.example file in version control, which is often used for providing configuration templates. * Refactor setup script and isolate wait logic The setup script now allows for an environment variable override for the gRPC URL and extracts the waiting logic for kwild readiness into a separate script. Additionally, the creation of the kwil_cli config file now utilizes the gRPC URL environment variable, improving configurability and adherence to the DRY principle. * Add clear-data task and update README Taskfile now includes a clear-data task to remove local Kwil data and Docker volumes. README instructions have been updated to reflect the new method of clearing data, and non-essential Docker Compose sections have been consolidated for clarity. * Refactor gRPC URL configuration in CI tests * Add vendor directory and go.work.sum to .dockerignore The .dockerignore has been updated to exclude the `vendor/` directory and the `go.work.sum` file. These changes ensure unnecessary directories and files are not included in the Docker build context, optimizing build performance. * Update go-isatty and go-tools to newer versions Upgraded github.com/mattn/go-isatty to v0.0.20 and golang.org/x/tools to v0.16.1. The update includes the removal of older module versions resulting in a cleaner module dependency graph. * Update dependencies in go.mod and go.sum Updated `go-spew` to `v1.1.2-0.20180830191138-d8f796af33cc`, `go-difflib` to `v1.0.1-0.20181226105442-5d4384ee4fb2`, and `golang.org/x/sys` to `v0.16.0`. Also added checksums in `go.sum` for these changes. * Update Docker image deployment logic Removed old build argument logic and streamlined the process of deploying Docker images on an EC2 instance. Added s3 asset for docker-compose file and adjusted service definition to use Docker Compose for service management. * Update instance to use Amazon Linux 2 Changed user data script to support Amazon Linux 2 AMI, ensuring compatibility with AWS environments. The Docker and AWS CLI installation process was adapted for yum package manager and the script now also includes docker-compose setup. Docker permissions are adjusted for the default ec2-user. * Add warning comment on environment variables A warning comment was added to getEnvStringsForService to alert maintainers about the lack of encryption in environment variables within the CloudFormation template and the potential access through service configuration files. The comment recommends switching to encryption if necessary. * Remove local uuidgen setup from CI It's not necessary anymore as data push is done on a docker image. * Streamline deployment workflow Remove unnecessary steps and adjust paths in `deploy-staging.yaml`. This change streamlines the deployment process by eliminating unused setup actions, simplifying Go commands, and updating paths to reflect the current infrastructure layout. * Add grep to Dockerfile dependencies The Dockerfile has been updated to include the installation of the grep tool. This addition is required to support the -P option in scripts, ensuring their proper functionality, particularly when retrying operations upon encountering nonce errors. * Update .gitignore * Update deployments/infra/go.mod Co-authored-by: Michael Buntarman <michaelboentarman@gmail.com> * update import path to reflect module name change * Remove unused test file for CDK main The file `cdk_main_test.go` was removed because it contained only commented-out code and unused tests. This should have no impact on the actual test suite as the file was not in use. --------- Co-authored-by: Michael Buntarman <michaelboentarman@gmail.com> * test: increase unit test coverage on extensions (#142) * docs: change Kwil to TSN documentation (#144) * fix(compose): compose locally seamlessly (#147) * feat: make kwil gateway binary available in ec2 instance (#141) * deployments/infra/cdk_main.go: make kwil gateway binary available in ec2 instance This PR extracts logic for creating ec2 role into its own function. Also it adds a boot up script to ec2 instance to fetch from s3 the kwil gateway binary. Resolves #109. * apply pr comments keep binary out of tmp folder * address pr comments change place of newName variable * address pr comment move binary to /usr/local/bin * address pr comments rewrite kwilGatewayBinaryScript --------- Co-authored-by: Raffael Campos <raffael.rcampos@gmail.com> Co-authored-by: VladislavSpassov <wladislawspassow@gmail.com>

deployments/infra/cdk_main.go: make kwil gateway binary available in …

c8391f5

…ec2 instance This PR extracts logic for creating ec2 role into its own function. Also it adds a boot up script to ec2 instance to fetch from s3 the kwil gateway binary. Resolves #109.

VladislavSpassov force-pushed the feat/fetch-kwil-gateway-binary-from-s3-new branch from 4a92e6b to c8391f5 Compare March 20, 2024 18:16

VladislavSpassov self-assigned this Mar 20, 2024

VladislavSpassov mentioned this pull request Mar 20, 2024

feat: make available kwil gateway binary in ec2 instance #140

Closed

11 tasks

apply pr comments

c81c225

keep binary out of tmp folder

VladislavSpassov changed the title ~~make kwil gateway binary available in ec2 instance~~ feat: make kwil gateway binary available in ec2 instance Mar 20, 2024

VladislavSpassov requested review from MicBun and outerlook March 20, 2024 18:50

VladislavSpassov marked this pull request as ready for review March 20, 2024 18:51

outerlook reviewed Mar 20, 2024

View reviewed changes

Comment thread deployments/infra/cdk_main.go Outdated

outerlook reviewed Mar 20, 2024

View reviewed changes

Comment thread deployments/infra/cdk_main.go Outdated

outerlook suggested changes Mar 20, 2024

View reviewed changes

VladislavSpassov added 2 commits March 20, 2024 22:55

address pr comments

1817e63

change place of newName variable

address pr comment

4c88d91

move binary to /usr/local/bin

VladislavSpassov requested a review from outerlook March 21, 2024 09:21

outerlook reviewed Mar 21, 2024

View reviewed changes

Comment thread deployments/infra/cdk_main.go

address pr comments

6307387

rewrite kwilGatewayBinaryScript

VladislavSpassov requested a review from outerlook March 21, 2024 12:28

outerlook approved these changes Mar 21, 2024

View reviewed changes

outerlook linked an issue Mar 21, 2024 that may be closed by this pull request

Problem: Unable to fetch the gateway binary (privately) to deploy its server #109

Closed

outerlook merged commit 8a2a3f1 into refactor/upgrade-zero-seven Mar 25, 2024

outerlook deleted the feat/fetch-kwil-gateway-binary-from-s3-new branch March 25, 2024 13:09

markholdex unassigned VladislavSpassov Apr 1, 2024

coderabbitai Bot mentioned this pull request Nov 20, 2025

chore: filter attestations by hash and payload #1260

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: make kwil gateway binary available in ec2 instance#141

feat: make kwil gateway binary available in ec2 instance#141
outerlook merged 5 commits into
refactor/upgrade-zero-sevenfrom
feat/fetch-kwil-gateway-binary-from-s3-new

VladislavSpassov commented Mar 20, 2024 •

edited

Loading

Uh oh!

outerlook commented Mar 20, 2024

Uh oh!

VladislavSpassov commented Mar 20, 2024

Uh oh!

Uh oh!

Uh oh!

outerlook left a comment •

edited

Loading

Uh oh!

VladislavSpassov commented Mar 20, 2024

Uh oh!

outerlook commented Mar 20, 2024 •

edited

Loading

Uh oh!

Uh oh!

outerlook left a comment

Uh oh!

markholdex commented Mar 22, 2024

Uh oh!

outerlook commented Mar 22, 2024

Uh oh!

VladislavSpassov commented Mar 25, 2024

Uh oh!

outerlook commented Mar 25, 2024

Uh oh!

VladislavSpassov commented Mar 25, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

VladislavSpassov commented Mar 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related Issue

Motivation and Context

How Has This Been Tested?

Types of changes

Checklist:

Checklist Explanation:

How to Review this PR:

Additional Information:

Uh oh!

outerlook commented Mar 20, 2024

Uh oh!

VladislavSpassov commented Mar 20, 2024

Uh oh!

Uh oh!

Uh oh!

outerlook left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

VladislavSpassov commented Mar 20, 2024

Uh oh!

outerlook commented Mar 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

outerlook left a comment

Choose a reason for hiding this comment

Uh oh!

markholdex commented Mar 22, 2024

Uh oh!

outerlook commented Mar 22, 2024

Uh oh!

VladislavSpassov commented Mar 25, 2024

Uh oh!

outerlook commented Mar 25, 2024

Uh oh!

VladislavSpassov commented Mar 25, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

VladislavSpassov commented Mar 20, 2024 •

edited

Loading

outerlook left a comment •

edited

Loading

outerlook commented Mar 20, 2024 •

edited

Loading