Issues with aws-containers subgenerator #8366
Comments
|
some logs |
|
Has anyone had luck with deploying using this? |
|
@deepu105 I have an issue when run jhipster aws-containers |
|
@ggotti : will you have time to have a look on this ticket, which is opened 7 months ago, plz ? |
|
Hello! I've been trying to get this subgenerator working today using a fairly simple JHipster project and have been having issues. My first issue was with the S3 bucket creation - if the process fails, the bucket is not removed and causes subsequent attempts to fail. Having solved this by adding more error feedback, I then ran into trouble with the CloudFormation stage failing. I am happy to put a little time into helping with this, but I am not sure where to begin; and I saw in issue #9685 that @pascalgrimaud was talking about removing this subgenerator from JHipster. So my questions are:
Thank you! |
|
@orbific : thanks a lot for taking time on this.
|
|
@pascalgrimaud: There are obvious issues with a generator that can't be regression tested which are probably wider than the scope of this issue. I agree this is an important piece of functionality, however, and worth supporting. There are a few issues I am aware of at present. Please let me know if you would like me to raise them as separate issues within github:
Invalid parameter at 'repositoryName' failed to satisfy constraint: 'must satisfy regular expression '(?:[a-z0-9]+(?:[.-][a-z0-9]+)/)[a-z0-9]+(?:[.-][a-z0-9]+)*'' Most of these issues were raised by @deepu105 in their initial report, and are still present in 6.0.1. I am going to continue working with this and see if I can get further. My failures are occurring at a much earlier stage than in the logs from September, so hopefully I can make some progress today. |
|
The good news is that I have now got the generator to work correctly, and deployed an application to AWS. The main problem is that the baseName property in .yo-rc.json is used in various places by the script without being changed to lower case. This causes an error with the docker tags, and also with the repositoryName. It should be simple to fix this however. There was also a minor issue with a stack trace/error message being thrown as part of a successful deployment. I am going to see if I can produce a fix for this issue - I can produce some tests for this without needing AWS access, as I can check the yml files produced. I have not tested that the deployment and scaling work correctly, and will move onto that once the generator is updated. |
|
@orbific You can start by providing a draft PR with only one thing fixed (for instance the non-deletion of the S3 bucket) and then go on with other fixes in this same PR. The issue we have with AWS is that our configuration isn't part of the free tier (and amazon doesn't give us credits) so we cannot test it... |
|
@wmarques Thank you for the feedback. It's going to be a week or so before I get more time to work on this, but that sounds like a great way to start. I think the best initial change would be to update the usages of the baseName that must be lowercase. |
|
@orbific Yes take your time no worries |
|
As noted above, I've created a pull request which fixes what was, for me, the breaking issue for this generator. With these changes made, I am able to run the command and create an AWS container deployment. I could not find any existing test cases for the generator. I am happy to write these if needed, but will need a little support (I am a Java developer, not a Javascript one). There are also a number of outstanding problems with the generator, as outlined above. The most important of these is that all executions of the script throw an error, even when successful. Should I create a new github issue for tracking this? |
|
I've added a $100 bug bounty here, as I know it's a huge work to sort this out. |
|
I'm reopening this ticket, following #9824 (comment) |
|
@orbific : Are you still working on this? If not I can take over. @pascalgrimaud @deepu105 : I have never worked with AWS, only with GCP but I guess, it should be somewhat similar in terms of the underlying concepts. So it might take me a bit of time to figure things out but if you guys are okay I can give it a shot. 😄 😄 |
|
@SudharakaP yes, please do take it over. I've not had chance to look at this properly and won't for a while. If you have any questions though, do send them my way and I'll see if I can help. |
|
@RizqonSidikMaulana : I got the same error your got in the beginning; however is your stack-trace redacted? Because in my one it gave me more information; specifically;
@deepu105 : Yes, for me after some minor changes the Monolith flow works correctly. I am currently working on testing other flows with this generator to do some improvements as stated in your original post. |
|
I was working on this over the weekend and there is quite a few problems with his subgenerator; I've created a pull request which resolves some of them. Firstly, in this pull request I've removed all aws related configuration from In @deepu105 's original post; I think I've solved 3 and 4 and maybe have added some improvements to 5 and 6. The creation of monoliths now seem to work and there's no error at the end as mentioned by @orbific and this alone should solve previously seen bugs as it wasn't correctly saving the aws configuration before. However there might be problems updating an existing deployment which I am working on right now. For now, this PR could be merged; I'll add another PR for solving all problems with the update flow; if that goes well I think we will have a working aws monolith deployment. The microservice flow is broken currently but that needs to be handled separately I think; as it's quite a bit of work to figure it out. I've tested this on an actual deployment. If anyone could test this out and post any other issues you guys see, I would really appreciate it; since this is a bit tricky to test due to the upload time (which is >15 mins). The monolith deployment doesn't take much money btw; for all my testing it only took around 5 dollars. |
|
I should add to my previous post that the basic update flow also works; I've just tested it as well. 😄 ✔ The S3 Bucket jhipstertest-1568035951000 has been created. ✔ base.template.yml has been updated to the S3 Bucket and can be found here: https://jhipstertest-1568035951000.s3.amazonaws.com/base.template.yml ✔ jhipstertest.template.yml has been updated to the S3 Bucket and can be found here: https://jhipstertest-1568035951000.s3.amazonaws.com/jhipstertest.template.yml ✔ Existing CloudFormation Stack for app jhipstertest Found ☁️ ✔ ECR Repository ID for app jhipstertest was found: jhipstertest/jhipstertest ✔ Password has been set in ASM Parameter: /jhipstertest/jhipstertest/spring.datasource.password ✔ ECR Repository URI for jhipstertest was found: 204529615325.dkr.ecr.us-west-2.amazonaws.com/jhipstertest/jhipstertest ✔ The Docker image was tagged: 204529615325.dkr.ecr.us-west-2.amazonaws.com/jhipstertest/jhipstertest:latest ✖ WARNING! Using --password via the CLI is insecure. Use --password-stdin. ✖ WARNING! Your password will be stored unencrypted in /home/sudharaka/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded ✔ Docker is now connected to your account in the region us-west-2. The push refers to repository [204529615325.dkr.ecr.us-west-2.amazonaws.com/jhipstertest/jhipstertest] fbe19b049210: Preparing 45a088dd656a: Preparing 921250086fd9: Preparing 5c99dd878f48: Preparing 5e600912c3fe: Preparing 0c513de401d6: Preparing c5597257aa29: Preparing 122be11ab4a2: Preparing 7beb13bce073: Preparing f7eae43028b3: Preparing 6cebf3abed5f: Preparing 0c513de401d6: Waiting c5597257aa29: Waiting 122be11ab4a2: Waiting 7beb13bce073: Waiting f7eae43028b3: Waiting 6cebf3abed5f: Waiting 5e600912c3fe: Layer already exists 45a088dd656a: Layer already exists fbe19b049210: Layer already exists 5c99dd878f48: Layer already exists 0c513de401d6: Layer already exists c5597257aa29: Layer already exists 122be11ab4a2: Layer already exists 7beb13bce073: Layer already exists f7eae43028b3: Layer already exists 6cebf3abed5f: Layer already exists 921250086fd9: Pushed latest: digest: sha256:a313021d0ee9cdc2a2e84c677de2cde0b9f5db3d41e93bf57019da75980c70f0 size: 2625 ✔ Image is now pushed to repository 204529615325.dkr.ecr.us-west-2.amazonaws.com/jhipstertest/jhipstertest:latest. ✔ Updating Existing CloudFormation Stack ☁️ 7:25:09 AM UPDATE IN_PROGRESS AWS::CloudFormation::Stackjhipstertest 7:25:16 AM UPDATE IN_PROGRESS AWS::RDS::DBCluster jhipstertest-jhipstertest-ZAQ3TPNP7T0W 7:26:17 AM UPDATE COMPLETE AWS::RDS::DBCluster jhipstertest-jhipstertest-ZAQ3TPNP7T0W 7:26:34 AM UPDATE IN_PROGRESS AWS::ECS::Service jhipstertest-jhipstertest-ZAQ3TPNP7T0W 7:28:35 AM UPDATE COMPLETE AWS::ECS::Service jhipstertest-jhipstertest-ZAQ3TPNP7T0W 7:33:36 AM UPDATE COMPLETE AWS::CloudFormation::Stackjhipstertest 7:33:33 AM UPDATE COMPLETE_CLEANUP_IN_PROGRESS AWS::CloudFormation::Stackjhipstertest-jhipstertest-ZAQ3TPNP7T0W 7:33:49 AM UPDATE COMPLETE AWS::CloudFormation::Stackjhipstertest 7:33:40 AM UPDATE COMPLETE AWS::CloudFormation::Stackjhipstertest-jhipstertest-ZAQ3TPNP7T0W The CloudFormation Stack jhipstertest has been updated ✔ Applications Accessible at Load Balancers: http://jhips-JHips-MWRDSDEZMNDL-1829635047.us-west-2.elb.amazonaws.com INFO! Congratulations, JHipster execution is complete! Configuration saved at awsConstants.json |
|
For this one, I will totally trust you @SudharakaP as, in the team, we don't have the credit and the bandwitch to maintain and to test in AWS. So big thanks to you I'm increasing the bounty so it will help you to have credit to test more. |
pascalgrimaud
added
$200
|
@pascalgrimaud : Thanks; keeping the bounty at $100 is fine with me as well; particularly because I know from earlier comments that we don't get funded by amazon. Anyways, there are lot of corner cases here and fixing all the problems will take more time (I haven't even touched the micro-service part). For some reason this sub-generator seems quite complex than the rest I've worked on. I haven't worked much on AWS and I'll read some more about this and see if we can rip this apart and improve this overall. It's quite weird that compared to the GAE deployments (which is quite seamless); deploying a simple monolith with this seems to be overly complex (at least for me). Or maybe given that we have the aws sub-generator which seems to be more in alignment with GAE for deploying simple monoliths; we should dedicate this sub-generator for micro-services only. Will think about it more....... |
|
@pascalgrimaud @deepu105 @PierreBesson : I believe we have a working monolith flow; I'll fix things as they come along (feel free to assign anything related to this sub-generator to me). On another note; I am going to try to get the micro-services flow working this week. I haven't worked much on micro-services in Jhipster; do you guys have a typical micro-service flow that you can recommend that I should test this with? I am asking for a base test case for micro-services so that we have consistency. |
|
The simplest way is to generate the whole architecture microservices, so you'll understand how it works.
|
|
@pascalgrimaud : Thanks much. 😄 |
|
@pascalgrimaud As part of this and GCP micro-service support, I am going through microservices generated by JHipster to get a thorough understanding. I've discovered that there are small errors in the PNG files/illustrations in our website. What did you guys use to create these PNGs? Are there any editable project files corresponding to these PNGs; if so let me know so that I can make these small edits as and when I go through them. I am talking about illustrations such as the following; https://github.com/jhipster/jhipster.github.io/blob/master/images/microservices_architecture_2.png 😄 cc: @jdubois |
|
I think I made them with Keynote, and I saved the original file in the same directory. |
|
@jdubois : There's one keynote file in there, but that only contains some illustrations (6 slides). 🤔 🤔 That's fine, there's no major changes I was gonna do to them, but if you do find the other .key files please do upload them. |
|
I'm not sure, maybe there is also some ASCII art made by @PierreBesson that he transformed into images with a script. |
|
This was done by @gmarziou using http://ditaa.sourceforge.net/ and http://asciiflow.com/ |
|
I've added the microservice flow for the Typical micro-service flow looks like this. Now this doesn't deploy the application to AWS but rather creates the EKS cluster and the ECRs and sets up everything necessary and directs the user to use the Kubernetes sub-generator to deploy it. sudharaka@Sudharaka-XPS-8700:~/IdeaProjects/GeneratedSamples/jhipstergateway$ jhipster aws-containers
INFO! Using JHipster version installed globally
INFO! Executing jhipster:aws-containers
INFO! Options: from-cli: true
This AWS generator will help you deploy your JHipster app as a Docker container on AWS.
✔ Docker is installed
✔ AWS credentials using profile default.
? Which *type* of application would you like to deploy? Microservice application
microservice
? Enter the root directory where your gateway(s) and microservices are located ../
3 applications found at /home/sudharaka/IdeaProjects/GeneratedSamples/
? Which applications do you want to include in your configuration? jhipstergateway, microservice1, microservice2
? Name of the EKS Cluster? jhipster
? What Kubernetes version would you like to use? 1.13
? Name of the Node Group? standard-workers
? On which region do you want to deploy? us-west-2
? Total number of nodes (for a static ASG)? 4
✔ Initialising Elastic Kubernetes Service (EKS)️ ...
[ℹ] using region us-west-2
[ℹ] setting availability zones to [us-west-2a us-west-2d us-west-2b]
[ℹ] subnets for us-west-2a - public:192.168.0.0/19 private:192.168.96.0/19
[ℹ] subnets for us-west-2d - public:192.168.32.0/19 private:192.168.128.0/19
[ℹ] subnets for us-west-2b - public:192.168.64.0/19 private:192.168.160.0/19
[ℹ] nodegroup "standard-workers" will use "ami-00b95829322267382" [AmazonLinux2/1.13]
[ℹ] using Kubernetes version 1.13
[ℹ] creating EKS cluster "jhipster" in "us-west-2" region
[ℹ] will create 2 separate CloudFormation stacks for cluster itself and the initial nodegroup
[ℹ] if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=us-west-2 --name=jhipster'
[ℹ] CloudWatch logging will not be enabled for cluster "jhipster" in "us-west-2"
[ℹ] you can enable it with 'eksctl utils update-cluster-logging --region=us-west-2 --name=jhipster'
[ℹ] 2 sequential tasks: { create cluster control plane "jhipster", create nodegroup "standard-workers" }
[ℹ] building cluster stack "eksctl-jhipster-cluster"
[ℹ] deploying stack "eksctl-jhipster-cluster"
[ℹ] building nodegroup stack "eksctl-jhipster-nodegroup-standard-workers"
[ℹ] --nodes-min=4 was set automatically for nodegroup standard-workers
[ℹ] --nodes-max=4 was set automatically for nodegroup standard-workers
[ℹ] deploying stack "eksctl-jhipster-nodegroup-standard-workers"
[✔] all EKS cluster resource for "jhipster" had been created
[✔] saved kubeconfig as "/home/sudharaka/.kube/config"
[ℹ] adding role "arn:aws:iam::204529615325:role/eksctl-jhipster-nodegroup-standar-NodeInstanceRole-18AYTV14ODD7H" to auth ConfigMap
[ℹ] nodegroup "standard-workers" has 0 node(s)
[ℹ] waiting for at least 4 node(s) to become ready in "standard-workers"
[ℹ] nodegroup "standard-workers" has 4 node(s)
[ℹ] node "ip-192-168-16-53.us-west-2.compute.internal" is ready
[ℹ] node "ip-192-168-34-228.us-west-2.compute.internal" is ready
[ℹ] node "ip-192-168-63-239.us-west-2.compute.internal" is ready
[ℹ] node "ip-192-168-80-107.us-west-2.compute.internal" is ready
[ℹ] kubectl command should work with "/home/sudharaka/.kube/config", try 'kubectl get nodes'
[✔] EKS cluster "jhipster" in "us-west-2" region is ready
✔ Initialising Elastic Repository Service (ERS) ...
{
"repository": {
"repositoryUri": "204529615325.dkr.ecr.us-west-2.amazonaws.com/microservice2",
"registryId": "204529615325",
"imageTagMutability": "MUTABLE",
"repositoryArn": "arn:aws:ecr:us-west-2:204529615325:repository/microservice2",
"repositoryName": "microservice2",
"createdAt": 1569467800.0
}
}
{
"repository": {
"repositoryUri": "204529615325.dkr.ecr.us-west-2.amazonaws.com/jhipstergateway",
"registryId": "204529615325",
"imageTagMutability": "MUTABLE",
"repositoryArn": "arn:aws:ecr:us-west-2:204529615325:repository/jhipstergateway",
"repositoryName": "jhipstergateway",
"createdAt": 1569467800.0
}
}
{
"repository": {
"repositoryUri": "204529615325.dkr.ecr.us-west-2.amazonaws.com/microservice1",
"registryId": "204529615325",
"imageTagMutability": "MUTABLE",
"repositoryArn": "arn:aws:ecr:us-west-2:204529615325:repository/microservice1",
"repositoryName": "microservice1",
"createdAt": 1569467800.0
}
}
✔ EKS and ECRs created. Please use the Kubernetes Sub-generator (jhipster kubernetes) to deploy.`
INFO! Congratulations, JHipster execution is complete!
Configuration saved at awsConstants.json
After that just going through the Kubernetes sub-generator will deploy it. I found this flow to be less error prone and quite consistent. Just for the record, I am just posting the rest of the deployment commands (all of which already documented in Kubenetes sub-generator) as well.
This deploys a two microservice+gateway application on AWS. I've done this deployment and here is the URL if you want to try it out; http://aeee38e48e01011e99a200ec93090b24-430535987.us-west-2.elb.amazonaws.com:8080/ (I'll take this down after a day or two so be quick 😄 ) Any comments or suggessions are welcome. There's lot more to be improved in this sub-generator; and I'll try to create more issues in the future. I am just adding basic micro-service support for this one so as to get this sub-generator to a workable level. 😄 😄 |
I have been playing with the AWS containers sub generator a lot these days and I find some issues there.
.yo-rc.jsonwhich is different from other cloud sub gens and could be confusingAssignPublicIpis enabled in the*AppService)The microservice flow is broken and I don't think it works anyway
The text was updated successfully, but these errors were encountered: