Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

How to limit the number of instances that get created in DigitalOcean #128

Closed
shairyar opened this issue Aug 1, 2018 · 5 comments
Closed

How to limit the number of instances that get created in DigitalOcean #128

shairyar opened this issue Aug 1, 2018 · 5 comments
Labels
bug connectors all issue about connectors
Milestone
4.0.0

Comments

@shairyar
Copy link

shairyar commented Aug 1, 2018 •
edited
Loading

Hi,

I am looking for a way to limit the number of droplets that get created in DigitalOcean so while looking at the documentaion I came across this link however that does not seem to work, right now I am seeing 10 droplets and i want to reduce that to 2 if possible, I am just playing around with this library

{
    "commander": {
        "password": "password"
    },
    "instance": {
        "port": 3128,
        "scaling": {
            "min": 1,
            "max": 2
        },
        "log": {
            "path": "/log"
        }
    },
    "providers": [
        {
            "type": "digitalocean",
            "token": "the-token-goes-here",
            "region": "SGP1",
            "size": "s-1vcpu-2gb",
            "sshKeyName": "ubuntu",
            "imageName": "name-of-image",
            "tags": "proxy,instance",
            "max": 2
        }
    ]
}

My understanding is by adding the max I should have been able to limit the droplets but I see no difference, is this the right please to add? What am i missing here?

I even tried changing the size to "size": "s-1vcpu-1gb" but that did not effect anything, still it is creating more instances that max specified
@fabienvauchelles
Copy link
Owner

Hello @shairyar ,

I cannot reproduce on DO. It is working fine for me.

Can you provide me a full logtrace ?
I'll try to understand what it is happening.

Thanks,
Fabien.

@shairyar
Copy link
Author

Hi @fabienvauchelles

This is what I see in my terminal, if there is anything you need me to provide please let me know.

I got it to work with amazon which was a relief.

Just so that you know i tried disabling the firewall and also with firewall enabled and I added the port 3128 as well on DigitalOcean.

I followed the steps mentioned in documentation very carefully and i went through the process multiple times just to make sure I did not miss anything

~/Sites/scraper scrapoxy start conf.json -d

2018-08-19T08:11:15.479Z - info: [Main] The selected providers are digitalocean/SGP1
2018-08-19T08:11:15.525Z - debug: [Main] listen
2018-08-19T08:11:15.528Z - info: [Commander] GUI is available at http://localhost:8889
2018-08-19T08:11:15.529Z - debug: [Manager] start
2018-08-19T08:11:15.530Z - info: Proxy is listening at http://localhost:8888
2018-08-19T08:11:25.532Z - debug: [Manager] checkInstances
::1 - - [19/Aug/2018:08:11:26 +0000] "GET / HTTP/1.1" 304 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:11:26 +0000] "GET /styles/vendor-7ee78e5317.css HTTP/1.1" 304 - "http://localhost:8889/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:11:26 +0000] "GET /styles/app-bc17cddbef.css HTTP/1.1" 304 - "http://localhost:8889/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:11:26 +0000] "GET /scripts/vendor-4dbec0db50.js HTTP/1.1" 304 - "http://localhost:8889/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:11:26 +0000] "GET /scripts/app-e4523c691d.js HTTP/1.1" 304 - "http://localhost:8889/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:11:27 +0000] "GET /assets/fonts/icomoon.ttf?-mts32b HTTP/1.1" 304 - "http://localhost:8889/styles/app-bc17cddbef.css" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:11:27 +0000] "GET /assets/fonts/OpenSans-Regular.woff HTTP/1.1" 304 - "http://localhost:8889/styles/app-bc17cddbef.css" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:11:27 +0000] "GET /api/scaling HTTP/1.1" 304 - "http://localhost:8889/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:11:27 +0000] "GET /api/instances HTTP/1.1" 200 2 "http://localhost:8889/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:11:27 +0000] "GET /assets/images/logo35h.png HTTP/1.1" 304 - "http://localhost:8889/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
2018-08-19T08:11:28.037Z - debug: [Manager] adjustInstances: required:1 / actual:0
2018-08-19T08:11:28.037Z - debug: [Manager] adjustInstances: add 1 instances
2018-08-19T08:11:28.038Z - debug: [ProviderDigitalOcean] createInstances: count=1
2018-08-19T08:11:35.536Z - debug: [Manager] checkInstances
2018-08-19T08:11:38.265Z - debug: [Manager] adjustInstances: required:1 / actual:0
2018-08-19T08:11:38.265Z - debug: [Manager] adjustInstances: add 1 instances
2018-08-19T08:11:38.265Z - debug: [ProviderDigitalOcean] createInstances: count=1
2018-08-19T08:11:45.539Z - debug: [Manager] checkInstances
2018-08-19T08:11:47.289Z - debug: [Manager] adjustInstances: required:1 / actual:0
2018-08-19T08:11:47.289Z - debug: [Manager] adjustInstances: add 1 instances
2018-08-19T08:11:47.289Z - debug: [ProviderDigitalOcean] createInstances: count=1
2018-08-19T08:11:55.539Z - debug: [Manager] checkInstances
2018-08-19T08:11:57.486Z - debug: [Manager] adjustInstances: required:1 / actual:0
2018-08-19T08:11:57.486Z - debug: [Manager] adjustInstances: add 1 instances
2018-08-19T08:11:57.486Z - debug: [ProviderDigitalOcean] createInstances: count=1
2018-08-19T08:12:05.539Z - debug: [Manager] checkInstances
2018-08-19T08:12:08.846Z - debug: [Manager] adjustInstances: required:1 / actual:0
2018-08-19T08:12:08.846Z - debug: [Manager] adjustInstances: add 1 instances
2018-08-19T08:12:08.847Z - debug: [ProviderDigitalOcean] createInstances: count=1
2018-08-19T08:12:15.542Z - debug: [Manager] checkInstances
::1 - - [19/Aug/2018:08:12:17 +0000] "GET /api/stats?retention=60000 HTTP/1.1" 200 - "http://localhost:8889/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
2018-08-19T08:12:17.804Z - debug: [Manager] adjustInstances: required:1 / actual:0
2018-08-19T08:12:17.804Z - debug: [Manager] adjustInstances: add 1 instances
2018-08-19T08:12:17.804Z - debug: [ProviderDigitalOcean] createInstances: count=1
::1 - - [19/Aug/2018:08:12:20 +0000] "GET /api/stats?retention=18000000 HTTP/1.1" 200 - "http://localhost:8889/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:12:21 +0000] "GET /api/stats?retention=86400000 HTTP/1.1" 200 - "http://localhost:8889/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
2018-08-19T08:12:25.546Z - debug: [Manager] checkInstances
2018-08-19T08:12:28.149Z - debug: [Manager] adjustInstances: required:1 / actual:0
2018-08-19T08:12:28.149Z - debug: [Manager] adjustInstances: add 1 instances
2018-08-19T08:12:28.149Z - debug: [ProviderDigitalOcean] createInstances: count=1
::1 - - [19/Aug/2018:08:12:33 +0000] "GET / HTTP/1.1" 304 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:12:33 +0000] "GET /styles/vendor-7ee78e5317.css HTTP/1.1" 304 - "http://localhost:8889/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:12:33 +0000] "GET /styles/app-bc17cddbef.css HTTP/1.1" 304 - "http://localhost:8889/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:12:33 +0000] "GET /scripts/vendor-4dbec0db50.js HTTP/1.1" 304 - "http://localhost:8889/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:12:33 +0000] "GET /scripts/app-e4523c691d.js HTTP/1.1" 304 - "http://localhost:8889/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:12:33 +0000] "GET /assets/fonts/OpenSans-Regular.woff HTTP/1.1" 304 - "http://localhost:8889/styles/app-bc17cddbef.css" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:12:33 +0000] "GET /assets/fonts/icomoon.ttf?-mts32b HTTP/1.1" 304 - "http://localhost:8889/styles/app-bc17cddbef.css" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:12:33 +0000] "GET /api/scaling HTTP/1.1" 304 - "http://localhost:8889/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:12:33 +0000] "GET /api/instances HTTP/1.1" 304 - "http://localhost:8889/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:12:33 +0000] "GET /assets/images/logo35h.png HTTP/1.1" 304 - "http://localhost:8889/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
::1 - - [19/Aug/2018:08:12:33 +0000] "GET /assets/images/logo32.png HTTP/1.1" 200 17953 "http://localhost:8889/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"
2018-08-19T08:12:35.548Z - debug: [Manager] checkInstances
2018-08-19T08:12:37.794Z - debug: [Manager] adjustInstances: required:1 / actual:0
2018-08-19T08:12:37.794Z - debug: [Manager] adjustInstances: add 1 instances
2018-08-19T08:12:37.794Z - debug: [ProviderDigitalOcean] createInstances: count=1
2018-08-19T08:12:45.551Z - debug: [Manager] checkInstances
2018-08-19T08:12:49.413Z - debug: [Manager] adjustInstances: required:1 / actual:0
2018-08-19T08:12:49.413Z - debug: [Manager] adjustInstances: add 1 instances
2018-08-19T08:12:49.413Z - debug: [ProviderDigitalOcean] createInstances: count=1
2018-08-19T08:12:55.556Z - debug: [Manager] checkInstances
2018-08-19T08:12:58.958Z - debug: [Manager] adjustInstances: required:1 / actual:0
2018-08-19T08:12:58.958Z - debug: [Manager] adjustInstances: add 1 instances
2018-08-19T08:12:58.958Z - debug: [ProviderDigitalOcean] createInstances: count=1
2018-08-19T08:13:05.559Z - debug: [Manager] checkInstances
^C2018-08-19T08:13:06.842Z - debug: [Main] shutdown
2018-08-19T08:13:06.842Z - debug: [Master] shutdown
2018-08-19T08:13:06.843Z - debug: [Manager] stop
2018-08-19T08:13:06.844Z - info: [Main] All instances are stopped.`

@marcelgwerder
Copy link

I'm experiencing the same problem with DO... Scrapoxy seems to keep adding more and more droplets (way more than my limit) and it also is unable to stop them in the end, I had to stop all of them manually.

@fabienvauchelles fabienvauchelles added the connectors all issue about connectors label Oct 4, 2023
@fabienvauchelles
Copy link
Owner

corrected in 4.0.0

@fabienvauchelles fabienvauchelles added this to the 4.0.0 milestone Oct 4, 2023
@fabienvauchelles
Copy link
Owner

Hey there! 🌟 Exciting news! Scrapoxy 4 is ready to rock 🚀. Check it out at Scrapoxy.io (explore the "get started" guide, deployment documentation, and more 📚).
I can't wait to hear your feedback on this new version! Send me your coolest screenshots with as many proxies as possible! 📸💻
Join the Discord community if you have any questions or just want to chat. You can also open a GitHub issue for any bug or feature request 🐞✨.
See you soon! 😎
Fabien

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug connectors all issue about connectors
Projects
None yet
Milestone
4.0.0
Development

No branches or pull requests
3 participants
@fabienvauchelles @marcelgwerder @shairyar