fix(compose): compose locally seamlessly#147
Merged
Merged
Conversation
MicBun
added a commit
that referenced
this pull request
Mar 28, 2024
* refactor: sync repo with kwil v0.7 * merge conflic * merge conflic * resolve conflicts * add .env to ignore * fix whitelist.go * fix typo * fix: setup-scripts (#127) * Delete .idea directory * fix(refactor-0.7): working date-range (#128) * add develop scripts * Refactor getValue and related methods, fixed to query with engine Context parameter was removed to streamline the method calls within the basestream module. The dbid argument was replaced by using scope directly, resulting in more concise and consistent function signatures across the module. * Refactor test suite to use new query execution interface Refactored `basestream_test.go` replacing the context and app setup with a `ProcedureContext` and a mock engine using a new query execution interface. Improved function calls to align with changes and updated test assertions to work with the revised structure and methods. * Add debug build target to Taskfile Implemented a new task `build:debug` for compiling the kwild binary with debug flags. This allows for better debugging experiences by disabling optimizations and inlining. * Update stream IDs in composed.kf Changed stream identifiers to use generic `/stream_b` and `/stream_a` instead of specific external service references. This update simplifies the development process by using placeholder values rather than actual external service streams. * Refactor build and deploy tasks in documentation Consolidated various script calls under single `task` commands for simplicity and readability. Updated the database seed section to reflect changes in order and improved the database query examples for consistency with stream_b. * remove unnecessary transform script * fix(refactor-0.7): working ci and dockerfile (#131) * fix(refactor-0.7): working ci and dockerfile * add docker compose * using setup for ci * hardcoded private key for seeding * revert run acceptance-test on draft * edit readme * put kwil-cli into .build * fix missing .. on database add primitive and hardcode whitelist wallets * add whitelist * add white list wallet * Update .github/workflows/ci.yaml Co-authored-by: Raffael Campos <raffael.rcampos@gmail.com> * replace WHITELIST_WALLETS --------- Co-authored-by: Raffael Campos <raffael.rcampos@gmail.com> * test(refactor-07): enable unit test on ci (#139) feat: enable unit test on ci * fix(refactor-0.7): make CDK infra compatible to latest dockerfile (#136) * Add infrastructure and CDK configuration for Go project Add new CDK configuration files, Go modules, Docker and Git ignore files, and placeholder Go tests for a Go project infrastructure setup. Adjust a docker file reference in `cdk_main.go`. * Remove cdk.iml and update .gitignore Removed the cdk.iml file from version control and added it to .gitignore to prevent it from being tracked. This ensures IDE-specific files do not clutter the repository and maintains a clean workspace for all collaborators. * Add Docker setup for TSN data push Added Dockerfile and updated CI and docker-compose configs to establish the environment for pushing TSN data. The CI workflow is adjusted with Python setup changes and additional steps to verify the compose status. Docker compose configuration now includes persistence and network definitions. * Update .gitignore to ignore all .env files except example The .gitignore file has been modified to ignore any files with the .env prefix. An exception has been added to keep the .env.example file in version control, which is often used for providing configuration templates. * Refactor setup script and isolate wait logic The setup script now allows for an environment variable override for the gRPC URL and extracts the waiting logic for kwild readiness into a separate script. Additionally, the creation of the kwil_cli config file now utilizes the gRPC URL environment variable, improving configurability and adherence to the DRY principle. * Add clear-data task and update README Taskfile now includes a clear-data task to remove local Kwil data and Docker volumes. README instructions have been updated to reflect the new method of clearing data, and non-essential Docker Compose sections have been consolidated for clarity. * Refactor gRPC URL configuration in CI tests * Add vendor directory and go.work.sum to .dockerignore The .dockerignore has been updated to exclude the `vendor/` directory and the `go.work.sum` file. These changes ensure unnecessary directories and files are not included in the Docker build context, optimizing build performance. * Update go-isatty and go-tools to newer versions Upgraded github.com/mattn/go-isatty to v0.0.20 and golang.org/x/tools to v0.16.1. The update includes the removal of older module versions resulting in a cleaner module dependency graph. * Update dependencies in go.mod and go.sum Updated `go-spew` to `v1.1.2-0.20180830191138-d8f796af33cc`, `go-difflib` to `v1.0.1-0.20181226105442-5d4384ee4fb2`, and `golang.org/x/sys` to `v0.16.0`. Also added checksums in `go.sum` for these changes. * Update Docker image deployment logic Removed old build argument logic and streamlined the process of deploying Docker images on an EC2 instance. Added s3 asset for docker-compose file and adjusted service definition to use Docker Compose for service management. * Update instance to use Amazon Linux 2 Changed user data script to support Amazon Linux 2 AMI, ensuring compatibility with AWS environments. The Docker and AWS CLI installation process was adapted for yum package manager and the script now also includes docker-compose setup. Docker permissions are adjusted for the default ec2-user. * Add warning comment on environment variables A warning comment was added to getEnvStringsForService to alert maintainers about the lack of encryption in environment variables within the CloudFormation template and the potential access through service configuration files. The comment recommends switching to encryption if necessary. * Update .gitignore * Update deployments/infra/go.mod Co-authored-by: Michael Buntarman <michaelboentarman@gmail.com> * update import path to reflect module name change * Remove unused test file for CDK main The file `cdk_main_test.go` was removed because it contained only commented-out code and unused tests. This should have no impact on the actual test suite as the file was not in use. --------- Co-authored-by: Michael Buntarman <michaelboentarman@gmail.com> * fix(refactor-0.7): Working CDK (#132) * Add infrastructure and CDK configuration for Go project Add new CDK configuration files, Go modules, Docker and Git ignore files, and placeholder Go tests for a Go project infrastructure setup. Adjust a docker file reference in `cdk_main.go`. * Remove cdk.iml and update .gitignore Removed the cdk.iml file from version control and added it to .gitignore to prevent it from being tracked. This ensures IDE-specific files do not clutter the repository and maintains a clean workspace for all collaborators. * Add Docker setup for TSN data push Added Dockerfile and updated CI and docker-compose configs to establish the environment for pushing TSN data. The CI workflow is adjusted with Python setup changes and additional steps to verify the compose status. Docker compose configuration now includes persistence and network definitions. * Update .gitignore to ignore all .env files except example The .gitignore file has been modified to ignore any files with the .env prefix. An exception has been added to keep the .env.example file in version control, which is often used for providing configuration templates. * Refactor setup script and isolate wait logic The setup script now allows for an environment variable override for the gRPC URL and extracts the waiting logic for kwild readiness into a separate script. Additionally, the creation of the kwil_cli config file now utilizes the gRPC URL environment variable, improving configurability and adherence to the DRY principle. * Add clear-data task and update README Taskfile now includes a clear-data task to remove local Kwil data and Docker volumes. README instructions have been updated to reflect the new method of clearing data, and non-essential Docker Compose sections have been consolidated for clarity. * Refactor gRPC URL configuration in CI tests * Add vendor directory and go.work.sum to .dockerignore The .dockerignore has been updated to exclude the `vendor/` directory and the `go.work.sum` file. These changes ensure unnecessary directories and files are not included in the Docker build context, optimizing build performance. * Update go-isatty and go-tools to newer versions Upgraded github.com/mattn/go-isatty to v0.0.20 and golang.org/x/tools to v0.16.1. The update includes the removal of older module versions resulting in a cleaner module dependency graph. * Update dependencies in go.mod and go.sum Updated `go-spew` to `v1.1.2-0.20180830191138-d8f796af33cc`, `go-difflib` to `v1.0.1-0.20181226105442-5d4384ee4fb2`, and `golang.org/x/sys` to `v0.16.0`. Also added checksums in `go.sum` for these changes. * Update Docker image deployment logic Removed old build argument logic and streamlined the process of deploying Docker images on an EC2 instance. Added s3 asset for docker-compose file and adjusted service definition to use Docker Compose for service management. * Update instance to use Amazon Linux 2 Changed user data script to support Amazon Linux 2 AMI, ensuring compatibility with AWS environments. The Docker and AWS CLI installation process was adapted for yum package manager and the script now also includes docker-compose setup. Docker permissions are adjusted for the default ec2-user. * Add warning comment on environment variables A warning comment was added to getEnvStringsForService to alert maintainers about the lack of encryption in environment variables within the CloudFormation template and the potential access through service configuration files. The comment recommends switching to encryption if necessary. * Remove local uuidgen setup from CI It's not necessary anymore as data push is done on a docker image. * Streamline deployment workflow Remove unnecessary steps and adjust paths in `deploy-staging.yaml`. This change streamlines the deployment process by eliminating unused setup actions, simplifying Go commands, and updating paths to reflect the current infrastructure layout. * Add grep to Dockerfile dependencies The Dockerfile has been updated to include the installation of the grep tool. This addition is required to support the -P option in scripts, ensuring their proper functionality, particularly when retrying operations upon encountering nonce errors. * Update .gitignore * Update deployments/infra/go.mod Co-authored-by: Michael Buntarman <michaelboentarman@gmail.com> * update import path to reflect module name change * Remove unused test file for CDK main The file `cdk_main_test.go` was removed because it contained only commented-out code and unused tests. This should have no impact on the actual test suite as the file was not in use. --------- Co-authored-by: Michael Buntarman <michaelboentarman@gmail.com> * test: increase unit test coverage on extensions (#142) * docs: change Kwil to TSN documentation (#144) * fix(compose): compose locally seamlessly (#147) * feat: make kwil gateway binary available in ec2 instance (#141) * deployments/infra/cdk_main.go: make kwil gateway binary available in ec2 instance This PR extracts logic for creating ec2 role into its own function. Also it adds a boot up script to ec2 instance to fetch from s3 the kwil gateway binary. Resolves #109. * apply pr comments keep binary out of tmp folder * address pr comments change place of newName variable * address pr comment move binary to /usr/local/bin * address pr comments rewrite kwilGatewayBinaryScript --------- Co-authored-by: Raffael Campos <raffael.rcampos@gmail.com> Co-authored-by: VladislavSpassov <wladislawspassow@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Related Issue
This PR is not based on an issue, but rather on fixing only
https://github.com/truflation/tsn-db/pull/136/files#diff-b30f5561a0300bfb88105c4e800a4f9cefbf93f6a69a656ec23aae85d20e9b54R41
Motivation and Context
Previously, when running
docker composefor the second timepush-tsn-datawould fail because there was some typopush-tsn-data.dockerfile, this small PR aims to fix that, alongside adding a private key ontask compose, since it also broke thetask composecommand because of strict requirement of the private keyHow Has This Been Tested?
task composelocally twice: Run it once, stop it, then run it againTypes of changes
Checklist:
Checklist Explanation:
How to Review this PR:
Additional Information: