AWS missing groupID

[benpetsch]

I got an issue setting up scrapoxy on my aws/ec2

starting like recommended:

sudo docker run -e COMMANDER_PASSWORD='MY_PASSWORD' \
    -e PROVIDERS_AWSEC2_ACCESSKEYID='MY_KEY' \
     -e PROVIDERS_AWSEC2_SECRETACCESSKEY='MY_SECRET_KEY' \
     -it -p 7777:8888 -p 8889:8889 fabienvauchelles/scrapoxy

running into this issue:

2017-07-09T14:33:59.747Z - info: [Main] The selected provider is awsec2
2017-07-09T14:33:59.764Z - debug: [Main] listen
2017-07-09T14:33:59.770Z - info: [Commander] GUI is available at http://localhost:8889
2017-07-09T14:33:59.772Z - debug: [Manager] start
2017-07-09T14:33:59.773Z - info: Proxy is listening at http://localhost:8888
2017-07-09T14:34:09.774Z - debug: [Manager] checkInstances
2017-07-09T14:34:10.162Z - debug: [Manager] adjustInstances: required:1 / actual:0
2017-07-09T14:34:10.163Z - debug: [Manager] adjustInstances: add 1 instances
2017-07-09T14:34:10.163Z - debug: [ProviderAWSEC2] createInstances: count=1
2017-07-09T14:34:10.446Z - debug: [ProviderAWSEC2] createInstances: actualCount=0
2017-07-09T14:34:11.038Z - error: [Manager] Error: Cannot update or adjust instances: InvalidParameterValue: Value () for parameter groupId is invalid. The value cannot be empty
    at Request.extractError (/usr/lib/node_modules/scrapoxy/node_modules/aws-sdk/lib/services/ec2.js:50:35)
    at Request.callListeners (/usr/lib/node_modules/scrapoxy/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
    at Request.emit (/usr/lib/node_modules/scrapoxy/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
    at Request.emit (/usr/lib/node_modules/scrapoxy/node_modules/aws-sdk/lib/request.js:615:14)
    at Request.transition (/usr/lib/node_modules/scrapoxy/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/usr/lib/node_modules/scrapoxy/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /usr/lib/node_modules/scrapoxy/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/usr/lib/node_modules/scrapoxy/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/usr/lib/node_modules/scrapoxy/node_modules/aws-sdk/lib/request.js:617:12)
    at Request.callListeners (/usr/lib/node_modules/scrapoxy/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
    at Request.emit (/usr/lib/node_modules/scrapoxy/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
    at Request.emit (/usr/lib/node_modules/scrapoxy/node_modules/aws-sdk/lib/request.js:615:14)
    at Request.transition (/usr/lib/node_modules/scrapoxy/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/usr/lib/node_modules/scrapoxy/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /usr/lib/node_modules/scrapoxy/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/usr/lib/node_modules/scrapoxy/node_modules/aws-sdk/lib/request.js:38:9)

Did I missconfigure the command?

Looks like aws denies request for missing security groupId

[zatarus]

+1

[kc1]

+1

[kc1]

Guys, I set up the project in the N. Virginia region because I could not access the eu west 1 region recommended in the quickstart. I think I'm having the same problem, I set it up using the docker command, but no instances are starting. Did you set it up in the eu west 1 region or somewhere else?

[fabienvauchelles]

Hello kc1,

Did you try to copy the AMI as recommended in the documentation? (http://docs.scrapoxy.io/en/master/standard/providers/awsec2/copy_ami_to_region/index.html)

AWS doesn't share AMI between region. I provide AMI for some regions but if it doesn't exist, you have to copy one yourself.

Fabien

[fabienvauchelles]

Hello benpetsch,

2 questions :

1/ do you use the eu-west1 region like in the documentation?
2/ do you have created security group and AMI in the same region?

Fabien

[kc1]

Hi Fabien, First, thank you for making this project available . Even though I don't have it working yet , but from what I can see it looks fantastic! In answer to your question, I set the project up in N. Virginia, because I could not get access to eu-west1region. I have a request into amazon to allow me to access to eu-west1. Hopefully they will allow it. If they don't, is there a way to deploy scrapoxy to another ec2 region directly? Thanks again, KC

…

On Sat, Jul 15, 2017 at 1:39 PM, Fabien Vauchelles ***@***.*** > wrote: Hello benpetsch, 2 questions : 1/ do you use the eu-west1 region like in the documentation? 2/ do you have created security group and AMI in the same region? Fabien — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#70 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABqjLn99hJLdCEJ7pr5xcVQzPyyvr68hks5sOPlugaJpZM4OSFKh> .

[fabienvauchelles]

Hello,

Of course you can do that ;p You have to copy the ami into your region. Check the link below, it will help you to do that.

Fabien

[kc1]

Hi Fabien, I was able to get it set up using docker in the eu-west1 region. Thank you.

[fabienvauchelles]

Nice!

[hotrush]

Hi @fabienvauchelles. Met the same issue, seems there was a toubles with security groups, we define in config any group name, but aws trying to assign default security group and vpc subnet. So, for me helps next config:

            "instance": {
                "InstanceType": "t2.nano",
                "ImageId": "ami-06220275",
                "SecurityGroupIds": [
                    "sg-xxxxxxxxxxx"
                ],
                "SubnetId": "subnet-xxxxxxxx"
            }

Defined manually security group and vpc subnet ids, ref https://github.com/aws/aws-sdk-js/blob/b8207ee91ae9d833b714db872e2a8d3923f4ab92/apis/ec2-2016-11-15.min.json#L8454

But i met another problem:

2017-08-23T10:38:28.057Z - debug: [Instance/i-04abbf60fbefa3b05] checkAlive: false / -
/usr/local/lib/node_modules/scrapoxy/server/common/pinger/index.js:19
        throw new Error('[ping] should have hostname and port');
        ^

Error: [ping] should have hostname and port
    at Object.ping (/usr/local/lib/node_modules/scrapoxy/server/common/pinger/index.js:19:15)
    at Timeout.self._checkAliveTimeout.setInterval [as _onTimeout] (/usr/local/lib/node_modules/scrapoxy/server/proxies/manager/instance.js:39:28)
    at ontimeout (timers.js:386:14)
    at tryOnTimeout (timers.js:250:5)
    at Timer.listOnTimeout (timers.js:214:5)

So, you are also need to seet this checkbox fot your VPC's subnet