fix(refactor-0.7): working date-range#128
Merged
Merged
Conversation
Context parameter was removed to streamline the method calls within the basestream module. The dbid argument was replaced by using scope directly, resulting in more concise and consistent function signatures across the module.
Refactored `basestream_test.go` replacing the context and app setup with a `ProcedureContext` and a mock engine using a new query execution interface. Improved function calls to align with changes and updated test assertions to work with the revised structure and methods.
Implemented a new task `build:debug` for compiling the kwild binary with debug flags. This allows for better debugging experiences by disabling optimizations and inlining.
Changed stream identifiers to use generic `/stream_b` and `/stream_a` instead of specific external service references. This update simplifies the development process by using placeholder values rather than actual external service streams.
Consolidated various script calls under single `task` commands for simplicity and readability. Updated the database seed section to reflect changes in order and improved the database query examples for consistency with stream_b.
MicBun
reviewed
Mar 18, 2024
MicBun
approved these changes
Mar 18, 2024
11 tasks
MicBun
added a commit
that referenced
this pull request
Mar 28, 2024
* refactor: sync repo with kwil v0.7 * merge conflic * merge conflic * resolve conflicts * add .env to ignore * fix whitelist.go * fix typo * fix: setup-scripts (#127) * Delete .idea directory * fix(refactor-0.7): working date-range (#128) * add develop scripts * Refactor getValue and related methods, fixed to query with engine Context parameter was removed to streamline the method calls within the basestream module. The dbid argument was replaced by using scope directly, resulting in more concise and consistent function signatures across the module. * Refactor test suite to use new query execution interface Refactored `basestream_test.go` replacing the context and app setup with a `ProcedureContext` and a mock engine using a new query execution interface. Improved function calls to align with changes and updated test assertions to work with the revised structure and methods. * Add debug build target to Taskfile Implemented a new task `build:debug` for compiling the kwild binary with debug flags. This allows for better debugging experiences by disabling optimizations and inlining. * Update stream IDs in composed.kf Changed stream identifiers to use generic `/stream_b` and `/stream_a` instead of specific external service references. This update simplifies the development process by using placeholder values rather than actual external service streams. * Refactor build and deploy tasks in documentation Consolidated various script calls under single `task` commands for simplicity and readability. Updated the database seed section to reflect changes in order and improved the database query examples for consistency with stream_b. * remove unnecessary transform script * fix(refactor-0.7): working ci and dockerfile (#131) * fix(refactor-0.7): working ci and dockerfile * add docker compose * using setup for ci * hardcoded private key for seeding * revert run acceptance-test on draft * edit readme * put kwil-cli into .build * fix missing .. on database add primitive and hardcode whitelist wallets * add whitelist * add white list wallet * Update .github/workflows/ci.yaml Co-authored-by: Raffael Campos <raffael.rcampos@gmail.com> * replace WHITELIST_WALLETS --------- Co-authored-by: Raffael Campos <raffael.rcampos@gmail.com> * test(refactor-07): enable unit test on ci (#139) feat: enable unit test on ci * fix(refactor-0.7): make CDK infra compatible to latest dockerfile (#136) * Add infrastructure and CDK configuration for Go project Add new CDK configuration files, Go modules, Docker and Git ignore files, and placeholder Go tests for a Go project infrastructure setup. Adjust a docker file reference in `cdk_main.go`. * Remove cdk.iml and update .gitignore Removed the cdk.iml file from version control and added it to .gitignore to prevent it from being tracked. This ensures IDE-specific files do not clutter the repository and maintains a clean workspace for all collaborators. * Add Docker setup for TSN data push Added Dockerfile and updated CI and docker-compose configs to establish the environment for pushing TSN data. The CI workflow is adjusted with Python setup changes and additional steps to verify the compose status. Docker compose configuration now includes persistence and network definitions. * Update .gitignore to ignore all .env files except example The .gitignore file has been modified to ignore any files with the .env prefix. An exception has been added to keep the .env.example file in version control, which is often used for providing configuration templates. * Refactor setup script and isolate wait logic The setup script now allows for an environment variable override for the gRPC URL and extracts the waiting logic for kwild readiness into a separate script. Additionally, the creation of the kwil_cli config file now utilizes the gRPC URL environment variable, improving configurability and adherence to the DRY principle. * Add clear-data task and update README Taskfile now includes a clear-data task to remove local Kwil data and Docker volumes. README instructions have been updated to reflect the new method of clearing data, and non-essential Docker Compose sections have been consolidated for clarity. * Refactor gRPC URL configuration in CI tests * Add vendor directory and go.work.sum to .dockerignore The .dockerignore has been updated to exclude the `vendor/` directory and the `go.work.sum` file. These changes ensure unnecessary directories and files are not included in the Docker build context, optimizing build performance. * Update go-isatty and go-tools to newer versions Upgraded github.com/mattn/go-isatty to v0.0.20 and golang.org/x/tools to v0.16.1. The update includes the removal of older module versions resulting in a cleaner module dependency graph. * Update dependencies in go.mod and go.sum Updated `go-spew` to `v1.1.2-0.20180830191138-d8f796af33cc`, `go-difflib` to `v1.0.1-0.20181226105442-5d4384ee4fb2`, and `golang.org/x/sys` to `v0.16.0`. Also added checksums in `go.sum` for these changes. * Update Docker image deployment logic Removed old build argument logic and streamlined the process of deploying Docker images on an EC2 instance. Added s3 asset for docker-compose file and adjusted service definition to use Docker Compose for service management. * Update instance to use Amazon Linux 2 Changed user data script to support Amazon Linux 2 AMI, ensuring compatibility with AWS environments. The Docker and AWS CLI installation process was adapted for yum package manager and the script now also includes docker-compose setup. Docker permissions are adjusted for the default ec2-user. * Add warning comment on environment variables A warning comment was added to getEnvStringsForService to alert maintainers about the lack of encryption in environment variables within the CloudFormation template and the potential access through service configuration files. The comment recommends switching to encryption if necessary. * Update .gitignore * Update deployments/infra/go.mod Co-authored-by: Michael Buntarman <michaelboentarman@gmail.com> * update import path to reflect module name change * Remove unused test file for CDK main The file `cdk_main_test.go` was removed because it contained only commented-out code and unused tests. This should have no impact on the actual test suite as the file was not in use. --------- Co-authored-by: Michael Buntarman <michaelboentarman@gmail.com> * fix(refactor-0.7): Working CDK (#132) * Add infrastructure and CDK configuration for Go project Add new CDK configuration files, Go modules, Docker and Git ignore files, and placeholder Go tests for a Go project infrastructure setup. Adjust a docker file reference in `cdk_main.go`. * Remove cdk.iml and update .gitignore Removed the cdk.iml file from version control and added it to .gitignore to prevent it from being tracked. This ensures IDE-specific files do not clutter the repository and maintains a clean workspace for all collaborators. * Add Docker setup for TSN data push Added Dockerfile and updated CI and docker-compose configs to establish the environment for pushing TSN data. The CI workflow is adjusted with Python setup changes and additional steps to verify the compose status. Docker compose configuration now includes persistence and network definitions. * Update .gitignore to ignore all .env files except example The .gitignore file has been modified to ignore any files with the .env prefix. An exception has been added to keep the .env.example file in version control, which is often used for providing configuration templates. * Refactor setup script and isolate wait logic The setup script now allows for an environment variable override for the gRPC URL and extracts the waiting logic for kwild readiness into a separate script. Additionally, the creation of the kwil_cli config file now utilizes the gRPC URL environment variable, improving configurability and adherence to the DRY principle. * Add clear-data task and update README Taskfile now includes a clear-data task to remove local Kwil data and Docker volumes. README instructions have been updated to reflect the new method of clearing data, and non-essential Docker Compose sections have been consolidated for clarity. * Refactor gRPC URL configuration in CI tests * Add vendor directory and go.work.sum to .dockerignore The .dockerignore has been updated to exclude the `vendor/` directory and the `go.work.sum` file. These changes ensure unnecessary directories and files are not included in the Docker build context, optimizing build performance. * Update go-isatty and go-tools to newer versions Upgraded github.com/mattn/go-isatty to v0.0.20 and golang.org/x/tools to v0.16.1. The update includes the removal of older module versions resulting in a cleaner module dependency graph. * Update dependencies in go.mod and go.sum Updated `go-spew` to `v1.1.2-0.20180830191138-d8f796af33cc`, `go-difflib` to `v1.0.1-0.20181226105442-5d4384ee4fb2`, and `golang.org/x/sys` to `v0.16.0`. Also added checksums in `go.sum` for these changes. * Update Docker image deployment logic Removed old build argument logic and streamlined the process of deploying Docker images on an EC2 instance. Added s3 asset for docker-compose file and adjusted service definition to use Docker Compose for service management. * Update instance to use Amazon Linux 2 Changed user data script to support Amazon Linux 2 AMI, ensuring compatibility with AWS environments. The Docker and AWS CLI installation process was adapted for yum package manager and the script now also includes docker-compose setup. Docker permissions are adjusted for the default ec2-user. * Add warning comment on environment variables A warning comment was added to getEnvStringsForService to alert maintainers about the lack of encryption in environment variables within the CloudFormation template and the potential access through service configuration files. The comment recommends switching to encryption if necessary. * Remove local uuidgen setup from CI It's not necessary anymore as data push is done on a docker image. * Streamline deployment workflow Remove unnecessary steps and adjust paths in `deploy-staging.yaml`. This change streamlines the deployment process by eliminating unused setup actions, simplifying Go commands, and updating paths to reflect the current infrastructure layout. * Add grep to Dockerfile dependencies The Dockerfile has been updated to include the installation of the grep tool. This addition is required to support the -P option in scripts, ensuring their proper functionality, particularly when retrying operations upon encountering nonce errors. * Update .gitignore * Update deployments/infra/go.mod Co-authored-by: Michael Buntarman <michaelboentarman@gmail.com> * update import path to reflect module name change * Remove unused test file for CDK main The file `cdk_main_test.go` was removed because it contained only commented-out code and unused tests. This should have no impact on the actual test suite as the file was not in use. --------- Co-authored-by: Michael Buntarman <michaelboentarman@gmail.com> * test: increase unit test coverage on extensions (#142) * docs: change Kwil to TSN documentation (#144) * fix(compose): compose locally seamlessly (#147) * feat: make kwil gateway binary available in ec2 instance (#141) * deployments/infra/cdk_main.go: make kwil gateway binary available in ec2 instance This PR extracts logic for creating ec2 role into its own function. Also it adds a boot up script to ec2 instance to fetch from s3 the kwil gateway binary. Resolves #109. * apply pr comments keep binary out of tmp folder * address pr comments change place of newName variable * address pr comment move binary to /usr/local/bin * address pr comments rewrite kwilGatewayBinaryScript --------- Co-authored-by: Raffael Campos <raffael.rcampos@gmail.com> Co-authored-by: VladislavSpassov <wladislawspassow@gmail.com>
zolotokrylin
added a commit
that referenced
this pull request
Apr 9, 2024
* refactor: sync repo with kwil v0.7 * merge conflic * merge conflic * resolve conflicts * add .env to ignore * fix whitelist.go * fix typo * fix: setup-scripts (#127) * Delete .idea directory * fix(refactor-0.7): working date-range (#128) * add develop scripts * Refactor getValue and related methods, fixed to query with engine Context parameter was removed to streamline the method calls within the basestream module. The dbid argument was replaced by using scope directly, resulting in more concise and consistent function signatures across the module. * Refactor test suite to use new query execution interface Refactored `basestream_test.go` replacing the context and app setup with a `ProcedureContext` and a mock engine using a new query execution interface. Improved function calls to align with changes and updated test assertions to work with the revised structure and methods. * Add debug build target to Taskfile Implemented a new task `build:debug` for compiling the kwild binary with debug flags. This allows for better debugging experiences by disabling optimizations and inlining. * Update stream IDs in composed.kf Changed stream identifiers to use generic `/stream_b` and `/stream_a` instead of specific external service references. This update simplifies the development process by using placeholder values rather than actual external service streams. * Refactor build and deploy tasks in documentation Consolidated various script calls under single `task` commands for simplicity and readability. Updated the database seed section to reflect changes in order and improved the database query examples for consistency with stream_b. * remove unnecessary transform script * Add gateway directives and owner restrictions to actions Added `@kgw(authn='true')` to `get_all`, `get_index`, and `get_value` actions for authentication enforcement. Specified `owner` privilege for the `add_record` action to restrict access. * fix(refactor-0.7): working ci and dockerfile (#131) * fix(refactor-0.7): working ci and dockerfile * add docker compose * using setup for ci * hardcoded private key for seeding * revert run acceptance-test on draft * edit readme * put kwil-cli into .build * fix missing .. on database add primitive and hardcode whitelist wallets * add whitelist * add white list wallet * Update .github/workflows/ci.yaml Co-authored-by: Raffael Campos <raffael.rcampos@gmail.com> * replace WHITELIST_WALLETS --------- Co-authored-by: Raffael Campos <raffael.rcampos@gmail.com> * test(refactor-07): enable unit test on ci (#139) feat: enable unit test on ci * fix(refactor-0.7): make CDK infra compatible to latest dockerfile (#136) * Add infrastructure and CDK configuration for Go project Add new CDK configuration files, Go modules, Docker and Git ignore files, and placeholder Go tests for a Go project infrastructure setup. Adjust a docker file reference in `cdk_main.go`. * Remove cdk.iml and update .gitignore Removed the cdk.iml file from version control and added it to .gitignore to prevent it from being tracked. This ensures IDE-specific files do not clutter the repository and maintains a clean workspace for all collaborators. * Add Docker setup for TSN data push Added Dockerfile and updated CI and docker-compose configs to establish the environment for pushing TSN data. The CI workflow is adjusted with Python setup changes and additional steps to verify the compose status. Docker compose configuration now includes persistence and network definitions. * Update .gitignore to ignore all .env files except example The .gitignore file has been modified to ignore any files with the .env prefix. An exception has been added to keep the .env.example file in version control, which is often used for providing configuration templates. * Refactor setup script and isolate wait logic The setup script now allows for an environment variable override for the gRPC URL and extracts the waiting logic for kwild readiness into a separate script. Additionally, the creation of the kwil_cli config file now utilizes the gRPC URL environment variable, improving configurability and adherence to the DRY principle. * Add clear-data task and update README Taskfile now includes a clear-data task to remove local Kwil data and Docker volumes. README instructions have been updated to reflect the new method of clearing data, and non-essential Docker Compose sections have been consolidated for clarity. * Refactor gRPC URL configuration in CI tests * Add vendor directory and go.work.sum to .dockerignore The .dockerignore has been updated to exclude the `vendor/` directory and the `go.work.sum` file. These changes ensure unnecessary directories and files are not included in the Docker build context, optimizing build performance. * Update go-isatty and go-tools to newer versions Upgraded github.com/mattn/go-isatty to v0.0.20 and golang.org/x/tools to v0.16.1. The update includes the removal of older module versions resulting in a cleaner module dependency graph. * Update dependencies in go.mod and go.sum Updated `go-spew` to `v1.1.2-0.20180830191138-d8f796af33cc`, `go-difflib` to `v1.0.1-0.20181226105442-5d4384ee4fb2`, and `golang.org/x/sys` to `v0.16.0`. Also added checksums in `go.sum` for these changes. * Update Docker image deployment logic Removed old build argument logic and streamlined the process of deploying Docker images on an EC2 instance. Added s3 asset for docker-compose file and adjusted service definition to use Docker Compose for service management. * Update instance to use Amazon Linux 2 Changed user data script to support Amazon Linux 2 AMI, ensuring compatibility with AWS environments. The Docker and AWS CLI installation process was adapted for yum package manager and the script now also includes docker-compose setup. Docker permissions are adjusted for the default ec2-user. * Add warning comment on environment variables A warning comment was added to getEnvStringsForService to alert maintainers about the lack of encryption in environment variables within the CloudFormation template and the potential access through service configuration files. The comment recommends switching to encryption if necessary. * Update .gitignore * Update deployments/infra/go.mod Co-authored-by: Michael Buntarman <michaelboentarman@gmail.com> * update import path to reflect module name change * Remove unused test file for CDK main The file `cdk_main_test.go` was removed because it contained only commented-out code and unused tests. This should have no impact on the actual test suite as the file was not in use. --------- Co-authored-by: Michael Buntarman <michaelboentarman@gmail.com> * fix(refactor-0.7): Working CDK (#132) * Add infrastructure and CDK configuration for Go project Add new CDK configuration files, Go modules, Docker and Git ignore files, and placeholder Go tests for a Go project infrastructure setup. Adjust a docker file reference in `cdk_main.go`. * Remove cdk.iml and update .gitignore Removed the cdk.iml file from version control and added it to .gitignore to prevent it from being tracked. This ensures IDE-specific files do not clutter the repository and maintains a clean workspace for all collaborators. * Add Docker setup for TSN data push Added Dockerfile and updated CI and docker-compose configs to establish the environment for pushing TSN data. The CI workflow is adjusted with Python setup changes and additional steps to verify the compose status. Docker compose configuration now includes persistence and network definitions. * Update .gitignore to ignore all .env files except example The .gitignore file has been modified to ignore any files with the .env prefix. An exception has been added to keep the .env.example file in version control, which is often used for providing configuration templates. * Refactor setup script and isolate wait logic The setup script now allows for an environment variable override for the gRPC URL and extracts the waiting logic for kwild readiness into a separate script. Additionally, the creation of the kwil_cli config file now utilizes the gRPC URL environment variable, improving configurability and adherence to the DRY principle. * Add clear-data task and update README Taskfile now includes a clear-data task to remove local Kwil data and Docker volumes. README instructions have been updated to reflect the new method of clearing data, and non-essential Docker Compose sections have been consolidated for clarity. * Refactor gRPC URL configuration in CI tests * Add vendor directory and go.work.sum to .dockerignore The .dockerignore has been updated to exclude the `vendor/` directory and the `go.work.sum` file. These changes ensure unnecessary directories and files are not included in the Docker build context, optimizing build performance. * Update go-isatty and go-tools to newer versions Upgraded github.com/mattn/go-isatty to v0.0.20 and golang.org/x/tools to v0.16.1. The update includes the removal of older module versions resulting in a cleaner module dependency graph. * Update dependencies in go.mod and go.sum Updated `go-spew` to `v1.1.2-0.20180830191138-d8f796af33cc`, `go-difflib` to `v1.0.1-0.20181226105442-5d4384ee4fb2`, and `golang.org/x/sys` to `v0.16.0`. Also added checksums in `go.sum` for these changes. * Update Docker image deployment logic Removed old build argument logic and streamlined the process of deploying Docker images on an EC2 instance. Added s3 asset for docker-compose file and adjusted service definition to use Docker Compose for service management. * Update instance to use Amazon Linux 2 Changed user data script to support Amazon Linux 2 AMI, ensuring compatibility with AWS environments. The Docker and AWS CLI installation process was adapted for yum package manager and the script now also includes docker-compose setup. Docker permissions are adjusted for the default ec2-user. * Add warning comment on environment variables A warning comment was added to getEnvStringsForService to alert maintainers about the lack of encryption in environment variables within the CloudFormation template and the potential access through service configuration files. The comment recommends switching to encryption if necessary. * Remove local uuidgen setup from CI It's not necessary anymore as data push is done on a docker image. * Streamline deployment workflow Remove unnecessary steps and adjust paths in `deploy-staging.yaml`. This change streamlines the deployment process by eliminating unused setup actions, simplifying Go commands, and updating paths to reflect the current infrastructure layout. * Add grep to Dockerfile dependencies The Dockerfile has been updated to include the installation of the grep tool. This addition is required to support the -P option in scripts, ensuring their proper functionality, particularly when retrying operations upon encountering nonce errors. * Update .gitignore * Update deployments/infra/go.mod Co-authored-by: Michael Buntarman <michaelboentarman@gmail.com> * update import path to reflect module name change * Remove unused test file for CDK main The file `cdk_main_test.go` was removed because it contained only commented-out code and unused tests. This should have no impact on the actual test suite as the file was not in use. --------- Co-authored-by: Michael Buntarman <michaelboentarman@gmail.com> * deployments/infra/cdk_main.go: make kwil gateway binary available in ec2 instance This PR extracts logic for creating ec2 role into its own function. Also it adds a boot up script to ec2 instance to fetch from s3 the kwil gateway binary. Resolves #109. * apply pr comments keep binary out of tmp folder * test: increase unit test coverage on extensions (#142) * address pr comments change place of newName variable * address pr comment move binary to /usr/local/bin * address pr comments rewrite kwilGatewayBinaryScript * Add gateway Dockerfile and Compose configurations This commit adds a Dockerfile, Compose configurations and related configurations for standing up a gateway service. New files include the Dockerfile, Compose YAML, Nginx configuration, and base JSON configurations for the gateway. Also, .gitignore is updated to prevent tracking of the binary needed for the gateway. By setting up this gateway service, the application gets more scalable and manageable. * Add Kgw asset and update systemd service Added a new asset for KgGateway in the CDK stack and modified the associated systemd service. A new systemd service for starting/stopping the KgGateway has also been introduced. These changes aim to improve the management and isolation of services within the infrastructure. * Add gateway environment variables to deploy-staging workflow This commit introduces gateway-related environment variables to the GitHub Actions deploy-staging workflow. Additional variables such as SESSION_SECRET and CORS_ALLOWED_ORIGINS have been included, which will be used in the gateway compose setup. * Update gateway configuration for SSL support The gateway configuration was updated to support SSL certificates. Nginx configuration files were updated and Dockerfile arguments were expanded to include domain information. Specifically, 'DOMAIN_NAME' variable was introduced and used to define the new server block in default.conf.template for handling SSL traffic. This variable is also added in gateway-compose.yaml to allow customization during deployment. The old 'kgw.nginx.conf' file was removed as the configurations have been moved to the new 'nginx.conf' and 'default.conf.template' files. * Add support for AWS Nitro Enclave certificate manager and domain configurations This update allows the Nitro Enclaves ACM agent to manage certificates with nginx and adds support for domain configurations in AWS. It also creates an IAM role that associates an AWS Nitro Enclave certificate with an IAM role. Additionally, new TSN-DB are made accessible to these instances. * Update gateway configuration and enable HTTPS This commit includes two key modifications in the default.conf.template file. Firstly, it adds a new location block to forward incoming requests to kwil-gateway on port 8090. Secondly, it includes a server block to handle the upgrade from http to https by default, improving the security of the connections. * Add port mapping for 443 in gateway-compose.yaml A new line has been added to the gateway-compose.yaml file to allow the port 443 for HTTPS connections. This change will help in accepting additional secure requests through our gateway. * Add local certificate and setup for dev-gateway This commit introduces local certificate files for secure development. A new Taskfile.yml action 'setup:local-cert' has been added for trusting these local certificates. The dev-gateway has been configured for docker deployment by adding dev-gateway.dockerfile and dev-gateway-compose.yaml. A .gitignore file is updated to ignore the private binary. * Add local gateway testing instructions This commit adds a README to the dev-gateway deployment directory with instructions on how to test the Gateway locally. It includes guidelines about software requirements, steps for setup and a command to test the successful operation of the Gateway. * Update gateway configurations and dockerfile This commit contains updates to the gateway configurations. The DOMAIN_NAME now defaults to a secure HTTP protocol (https) and the SSL protocols now include TLSv1.3. Changes were also made on the nginx.conf and gateway dockerfile. In the Gateway Dockerfile, environment variables were adjusted, allowing for automatic configuration based on these settings. * Update gateway configuration and Dockerfile The gateway configuration, 'gateway-compose.yaml', no longer defaults to * for CORS, as its input must start with https or http. This is reflected in the updated Dockerfile as well. Furthermore, the 'backends' field in 'kgw.base-config.json' has been changed from a string to an array. * Update gateway version in cdk_main.go The path for the gateway in cdk_main.go has been updated to reference the newer version (kgw_v0.1.3.zip) of the package. This change ensures that the AWS instance reads the updated package from the specified bucket. * Update object path in cdk_main.go The object path in cdk_main.go was updated from "gateway/kgw_v0.1.3.zip" to "gateway/kgw-0.1.3.zip". This is to ensure proper path naming and version formatting. * Update object path in cdk_main.go The object path in cdk_main.go was updated from "gateway/kgw_v0.1.3.zip" to "gateway/kgw-0.1.3.zip". This is to ensure proper path naming and version formatting. * fix incorrect conflicts * fix incorrect conflicts * add missing import from conflict * Refactor domain name usage in code Moved the main domain name into a constant for easier maintenance and readability. This change impacts how hosted zones and deployment stages reference the domain name, now using shared constant instead of hardcoded strings. * Update MainDomain constant in domain_utils The MainDomain in the domain_utils.go file has been updated to point to the test environment. Now the constant points to "tsn.test.truflation.com", that's a necessary update to align with recent changes in our system architecture. * Add utility functions to update parameters in cdk_main.go Utility functions have been introduced to perform conversion of parameters string to a map and then update map values. This facilitates efficient parameter handling within AWS CDK setup. Specific functions include: ConvertParamsToMap, UpdateMapValues, and UpdateParamsWithImageName. * fix string expectations on construct id * Update gateway binary file path and remove smoke test The binary file path in the `move` command has been updated to include an extra 'kgw' in its destination. This change aligns with the expected location in the docker-compose file. Moreover, the smoke test command that uses curl has been removed. This deletion eliminates an unnecessary operation in the AWS Nitro Enclaves CLI setup. * Add new hosted zone to cdk.context.json The commit adds a new hosted zone with a specific ID and name to the cdk.context.json file. This is done under the account: 344375646931, domainName: tsn.test.truflation.com, and region: us-east-2. * Update SSL configurations and enable Certbot for certificates The SSL configurations have been modified across the gateway; the `pkcs11` engine is no longer used. Certificates are now obtained through Let's Encrypt using Certbot, which has been integrated into the deployment process for the gateway. The smoke test in the Dockerfile has also been updated. In the infrastructure code, functions to install Certbot on the instance and add DNS validation have been added. * Update gateway setup and handle instance creation This commit updates the gateway setup in the AWS CDK infrastructure code, removing the AWS Nitro Enclaves CLI installation and adding the installation of certbot for SSL certificate management. It also adjusts case handling during instance creation by appending the `true` command to ExecStart, ensuring that the service always starts regardless of docker-compose's exit code. Code for returning an elastic IP object upon the creation of an instance has been added. * remove acm enclave association * Remove SSL configuration from gateway The SSL configuration related to Let's Encrypt in the gateway service was removed. This includes removal of certificate bindings and SSL settings in the Nginx configuration. The gateway now listens on HTTP instead of HTTPS. * Refactor Nitro Enclave setup and IAM role association The Nitro Enclave setup and IAM role association code has been moved from the file domain_utils.go to a new file named nitro_enclave.go. This change provides better code structuring and separates the responsibilities of each file. The GetACMCertificate function has been updated to take a constructs.Construct type argument instead of awscdk.Stack type. * Implement use of ACM for SSL on CloudFront The code now implements AWS Certificate Manager for SSL certificate generation. This change primarily impacts CloudFront, as ACM will now handle certificate generation for that service. Previously, certificate functionality was handled through CertBot, which was removed due to limitations with rate and security implications. An ACM certificate function was created for this purpose. * Update gateway_utils to use new version of kgw binary The changes remove unused imports in the gateway_utils.go file and update the S3 paths to use a new version of the kgw binary (v0.2.0), replacing the previous version (0.1.3). Additionally, significant portions of code related to Certbot installation and configuration have been removed. * Add curl and jq to gateway Dockerfile and dynamically fetch chain_id This update adds curl and jq commands to the Dockerfile in the gateway deployment, facilitating a dynamic fetch of the 'chain_id' from the tsn-db API. As a result of this change, the static 'chain_id' is removed from the 'kgw.base-config.json' file. This optimizes container set up by reducing hardcoded configurations and enhancing flexibility. * Enable deployment of all stacks in staging The changes made in `.github/workflows/deploy-staging.yaml` now allow for the deployment of all CloudFormation stacks in the staging environment. Previously, the configuration was only deploying a single stack. The `--all` flag has been added to `cdk deploy` command to ensure all stacks get deployed. * Update CloudFront Distribution to disable caching The method `CloudfrontForEc2Instance` in `cloudfront.go` has been updated to disable caching at CloudFront level and forward all the headers to the instance. This change ensures that every request hits the EC2 instance directly without being cached at the edge locations. * Add help comment to CloudfrontForEc2Instance function * Add entrypoint.sh for gateways and update Dockerfiles Introduced an entrypoint.sh script for both gateway and dev-gateway folders. This script will handle setting the CORS_ARGS and CHAIN_ID environment variables. Dockerfiles have been updated to include this script and to remove the previously-existing variable setting logic. Also, fixed the kgw.base-config.json files by removing comma. * Add security note and update Cloudfront settings A reminder to further secure the instances by allowing only specific IPs has been added. Modifications have been made in the Cloudfront settings as well, including allowing all methods and enabling the Origin Request Policy for all viewers. * Remove cache movement step in deploy-staging workflow The cache movement step in deploy-staging.yaml workflow file was unnecessary and has been removed. * Update cache_to parameter in compose.yaml The parameter 'src' in 'cache_to' settings of 'tsn-db' and 'push-tsn-data' services in the docker compose file has been fixed, correctly using 'dest'. * Add secrets check to staging deployment workflow The staging deployment workflow has been updated to include a step that checks for the existence of required secrets. This additional step will ensure that the process fails early if any of the necessary secrets are missing. --------- Co-authored-by: MicBun <michaelboentarman@gmail.com> Co-authored-by: Vladislav Spasov <wladislawspassow@gmail.com> Co-authored-by: Vadim <1125014+zolotokrylin@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This ensures previously working behavior for querying date ranges and getting the last value before a date.
It also adds the minimum development scripts to help tackle issues individually.
Related Issue
Closes #121
Motivation and Context
How Has This Been Tested?
Execute query scripts from
develop_scripts.mdand see the expectation is matchedTypes of changes
Checklist:
Checklist Explanation:
How to Review this PR:
Additional Information: