How to use several network cards for scaling/avoiding request limits

[Jabbaxx]

Binance has a rate limit of 1200 requests per IP per minute.
There will be times that one of my strategies will hit that. As I want to run multiple strategies at the same time I want to use several IP addresses on one system. And then have one node process or node-binance-api use just one specific address. Then next nodejs/node-binace-api process use another IP.
I dont want to use several linux systems as that will increase overhead of management and updating of servers.
So how to do that ?
I save something here about export https_proxy.
So could it be
export https_proxy=http://ip:port
and then use squid ? (not tested)

[jaggedsoft]

yeah you would likely have to pay for some socks5 proxies
digitalocean servers come with 16 ipv6 ip's that you could bind as local interfaces but unfortunately binance requires ipv4 which is more expensive, so you're most likely going to have to pay for some proxies

[Jabbaxx]

Thanks for feedback.

So I understand it right, the node-binance-api will honor the "export https_proxy=http://ip:port" setting before starting node ?

Regarding Proxy, I like to have things in my control ;) and I think it can be solved quite cheap with one linux box from around 5-10€/month with 3 IP's (additional IP = 1€ ). Or a couple of boxes if the CPU load is higher.
You can run Squid on the linux box something like this
and then with "export https_proxy" you control which of the IP's are used.
I will try that out ;) I have already ordered and got delivered 2 root boxes and 2 extra IP's.

Regarding my prices. https://www.netcup.eu/ will provide you a VPS server for under 3€ per month or a root server 6-8€ per month. And then each IP is another 1€. you can get 8 IP's without any questions. After 8 IPs they have some ripe stuff to check.
So basically with 10€ per month (8€+2*1€) you already have 3 IP's you can use.
Disclaimer. I have no connction with netcup, just using them myself as they have good prices and never had problems. Support answers fast on questions. Dont blame me for anything ;)

[jaggedsoft]

Great info thanks for sharing friend :)

[Jabbaxx]

Thanks a lot for writing the node module , its great work :)

Looks like it works great with Squid :)
Have not load tested it yet, but the config is like this.
The fat IP's you need to adopt to your config/system

Allow the networks I have on this machine - check ifconfig/??? to find out the network you machine has

acl localnetA src 99.99.99.0/24

Extra IPs this machine has

acl extraIP01 src 99.99.99.10/32

http_access allow localhost
http_access allow localnetA
http_access allow extraIP01
http_access deny all
http_port 127.0.0.1:3128

Same IP as extraIP01 above, that is the address connections to binance will come from

tcp_outgoing_address 99.99.99.10
dns_v4_first on

then before starting node you run this command:
export https_proxy=http://localhost:3128

I hope someone can use this.

[din1369]

Thanks a lot for writing the node module , its great work :)

Looks like it works great with Squid :)
Have not load tested it yet, but the config is like this.
The fat IP's you need to adopt to your config/system

Allow the networks I have on this machine - check ifconfig/??? to find out the network you machine has

acl localnetA src 99.99.99.0/24

Extra IPs this machine has

acl extraIP01 src 99.99.99.10/32

http_access allow localhost
http_access allow localnetA
http_access allow extraIP01
http_access deny all
http_port 127.0.0.1:3128

Same IP as extraIP01 above, that is the address connections to binance will come from

tcp_outgoing_address 99.99.99.10
dns_v4_first on

then before starting node you run this command:
export https_proxy=http://localhost:3128

I hope someone can use this.

Hi, i try these steps as you mentioned but for me still not working.

how did you check ip rotation is working and did you do any code level changes to change api calling?

this is my squid config. please check and let me know if there is any issue.
acl localnet src 0.0.0.1-0.255.255.255 # RFC 1122 "this" network (LAN)
acl localnet src 10.0.0.0/8 # RFC 1918 local private network (LAN)
acl localnet src 100.64.0.0/10 # RFC 6598 shared address space (CGN)
acl localnet src 169.254.0.0/16 # RFC 3927 link-local (directly plugged) machines
acl localnet src 172.16.0.0/12 # RFC 1918 local private network (LAN)
acl localnet src 192.168.0.0/16 # RFC 1918 local private network (LAN)
acl localnet src fc00::/7 # RFC 4193 local private network range
acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines
acl localnetA src xxxx/29
acl extraIP01 src xxxx1/32
acl extraIP02 src xxxx2/32
acl extraIP03 src xxxx3/32
acl extraIP04 src xxxx4/32
acl extraIP05 src xxxx5/32

acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT

tcp_outgoing_address xxxx1
tcp_outgoing_address xxxx2
tcp_outgoing_address xxxx3
tcp_outgoing_address xxxx4
tcp_outgoing_address xxxx5

http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager

include /etc/squid/conf.d/*

http_access allow localhost

http_access allow localnetA
http_access allow extraIP01
http_access allow extraIP02
http_access allow extraIP03
http_access allow extraIP04
http_access allow extraIP05

http_access deny all
http_port 127.0.0.1:3128

dns_v4_first on

[MegaManSE]

I was able to get it to work with scrapoxy. I've heard of squid but never used it. Hope that helps.

…

On Tue, Jan 12, 2021 at 7:27 AM din1369 ***@***.***> wrote: Thanks a lot for writing the node module , its great work :) Looks like it works great with Squid :) Have not load tested it yet, but the config is like this. The *fat IP's* you need to adopt to your config/system Allow the networks I have on this machine - check ifconfig/??? to find out the network you machine has acl localnetA src *99.99.99.0/24 <http://99.99.99.0/24>* Extra IPs this machine has acl extraIP01 src *99.99.99.10/32 <http://99.99.99.10/32>* http_access allow localhost http_access allow localnetA http_access allow extraIP01 http_access deny all http_port 127.0.0.1:3128 Same IP as extraIP01 above, that is the address connections to binance will come from tcp_outgoing_address *99.99.99.10* dns_v4_first on then before starting node you run this command: export https_proxy=http://localhost:3128 I hope someone can use this. Hi, i try these steps as you mentioned but for me still not working. how did you check ip rotation is working and did you do any code level changes to change api calling? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#430 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAUBKEZW4POORXVRDGA2XKLSZQ52ZANCNFSM4NFXWECQ> .

-- Will Perone

[din1369]

@MegaManSE can you share instructions?
Thank you.

[MegaManSE]

binance[account.id] = require('node-binance-api')(); ... if (util.config.proxy) { options.proxy = { auth: { username: util.config.proxy.username, password: util.config.proxy.password }, host: util.config.proxy.host, port: util.config.proxy.port }; } binance[account.id].options(options, validate);

…

On Tue, Jan 12, 2021 at 10:28 AM din1369 ***@***.***> wrote: @MegaManSE <https://github.com/MegaManSE> can you share instructions? Thank you. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#430 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAUBKE25APJ7TPSX27YM3KLSZRTB3ANCNFSM4NFXWECQ> .

-- Will Perone

[Jabbaxx]

Hi, I am not sure what you mean with IP rotation. You will have to send the squid config file and describe what you did to be able to help. This instructions are meant to be added to the squid config file and then just do the export on shell level before starting node. Maybe you need to use http instead of https on the export proxy. Or better add both. I did not change the code in any way. It already supports proxy.

…

On Tue, 12 Jan 2021, 14:27 din1369, ***@***.***> wrote: Thanks a lot for writing the node module , its great work :) Looks like it works great with Squid :) Have not load tested it yet, but the config is like this. The *fat IP's* you need to adopt to your config/system Allow the networks I have on this machine - check ifconfig/??? to find out the network you machine has acl localnetA src *99.99.99.0/24 <http://99.99.99.0/24>* Extra IPs this machine has acl extraIP01 src *99.99.99.10/32 <http://99.99.99.10/32>* http_access allow localhost http_access allow localnetA http_access allow extraIP01 http_access deny all http_port 127.0.0.1:3128 Same IP as extraIP01 above, that is the address connections to binance will come from tcp_outgoing_address *99.99.99.10* dns_v4_first on then before starting node you run this command: export https_proxy=http://localhost:3128 I hope someone can use this. Hi, i try these steps as you mentioned but for me still not working. how did you check ip rotation is working and did you do any code level changes to change api calling? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#430 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIULVKECCFYIADQ6MAYUODTSZQ52XANCNFSM4NFXWECQ> .

[din1369]

@MegaManSE Thank you.

@Jabbaxx this is my current squid configs

acl localnet src 0.0.0.1-0.255.255.255 # RFC 1122 "this" network (LAN)
acl localnet src 10.0.0.0/8 # RFC 1918 local private network (LAN)
acl localnet src 100.64.0.0/10 # RFC 6598 shared address space (CGN)
acl localnet src 169.254.0.0/16 # RFC 3927 link-local (directly plugged) machines
acl localnet src 172.16.0.0/12 # RFC 1918 local private network (LAN)
acl localnet src 192.168.0.0/16 # RFC 1918 local private network (LAN)
acl localnet src fc00::/7 # RFC 4193 local private network range
acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines
acl localnetA src xxxx/29
acl extraIP01 src xxxx1/32
acl extraIP02 src xxxx2/32
acl extraIP03 src xxxx3/32
acl extraIP04 src xxxx4/32
acl extraIP05 src xxxx5/32

acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT

tcp_outgoing_address xxxx1
tcp_outgoing_address xxxx2
tcp_outgoing_address xxxx3
tcp_outgoing_address xxxx4
tcp_outgoing_address xxxx5

http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost manager
http_access deny manager

include /etc/squid/conf.d/*

http_access allow localhost

http_access allow localnetA
http_access allow extraIP01
http_access allow extraIP02
http_access allow extraIP03
http_access allow extraIP04
http_access allow extraIP05

http_access deny all
http_port 127.0.0.1:3128

dns_v4_first on

then I did export o shell aso

export https_proxy=http://localhost:3128
export http_proxy=http://localhost:3128

but I still get

body: '{"code":-1003,"msg":"Way too much request weight used; IP banned until 1610514306891. Please use the websocket for live updates to avoid bans."}'

please check configs and let me know if there is anything to change.

Thank you.

[Jabbaxx]

Ahhh, now it makes sense. I am quite sure that your Squid proxy is working. Otherwise your application would not work at all anymore. First of all, I am using 99.999% futures. So this all is about futures. it might work also on SPOT and margin. No idea. The issue is that you misunderstand how squid works. Squid does not do rotating use of all the IP's you configured. Possibly in your config it just uses the last of "tcp_outgoing_address" or the first. No idea. I guess you only have one application that you start with node. I have 12 (or more) different per server. And squid is actually running one time PER outgoing IP. Each squid as its own port. Like 3128 first squid , 3129 second squid and so on. Then on the start of the first node app i do : export https_proxy=http://localhost:3128 export http_proxy=http://localhost:3128 *on start of the second node app i do * export https_proxy=http://localhost:3129 export http_proxy=http://localhost:312 <http://localhost:3128/>9 I think now it is more clear how it works. (I am using different ports than this, but the basic idea is the same) You will have to run your node application one time per External IP you have and with *different export lines.* I have a more complex setup as I have squid running one time per port/external IP. But i just found a description doing it in one squid config: https://tastyplacement.com/squid-proxy-multiple-outgoing-ip-addresses If this config is not working for you, I can post how to do it my way, but I really recommend the above link as it is a lot easier as I did ;) Also its possible to control per user and other solutions surely also exists. google is your friend here ;) https://serverfault.com/questions/929594/squid-4-2-different-outbound-ip-for-each-client-user If you cannot run your application several times, then you must find another solution. If you find it, please post it here. BTW, you can monitor the Weight usage with exchange.usedWeight() and then slow down the usage when you get near the 2400 weight. The weight is 2400 now. exchange.futuresExchangeInfo() says 2400 weight or 1200 requests on my system: rateLimits: [{ rateLimitType: 'REQUEST_WEIGHT', interval: 'MINUTE', intervalNum: 1, limit: 2400 }, {rateLimitType: 'ORDERS', interval: 'MINUTE', intervalNum: 1, limit: 1200 } ], Correct me if I am wrong. Also I would recommend you to start using WebSocket data if you can. I have only 3 requests for each additional trade I do. First cancel the old TP order with REST, create a new market order with REST (user Websocket to wait for filled) and then create the new TP order with REST. Everything else is done via WebSocket. The Highest Weight I have had on any of my apps since I added the additional Squids is 750. It's always good to be in control and monitor things, so you know when they will soon break. Good luck. Kind regards Jabbaxx

…

On Wed, Jan 13, 2021 at 6:49 AM din1369 ***@***.***> wrote: @MegaManSE <https://github.com/MegaManSE> Thank you. @Jabbaxx <https://github.com/Jabbaxx> this is my current squid configs acl localnet src 0.0.0.1-0.255.255.255 # RFC 1122 "this" network (LAN) acl localnet src 10.0.0.0/8 # RFC 1918 local private network (LAN) acl localnet src 100.64.0.0/10 # RFC 6598 shared address space (CGN) acl localnet src 169.254.0.0/16 # RFC 3927 link-local (directly plugged) machines acl localnet src 172.16.0.0/12 # RFC 1918 local private network (LAN) acl localnet src 192.168.0.0/16 # RFC 1918 local private network (LAN) acl localnet src fc00::/7 # RFC 4193 local private network range acl localnet src fe80::/10 # RFC 4291 link-local (directly plugged) machines acl localnetA src xxxx/29 acl extraIP01 src xxxx1/32 acl extraIP02 src xxxx2/32 acl extraIP03 src xxxx3/32 acl extraIP04 src xxxx4/32 acl extraIP05 src xxxx5/32 acl SSL_ports port 443 acl Safe_ports port 80 # http acl Safe_ports port 21 # ftp acl Safe_ports port 443 # https acl Safe_ports port 70 # gopher acl Safe_ports port 210 # wais acl Safe_ports port 1025-65535 # unregistered ports acl Safe_ports port 280 # http-mgmt acl Safe_ports port 488 # gss-http acl Safe_ports port 591 # filemaker acl Safe_ports port 777 # multiling http acl CONNECT method CONNECT tcp_outgoing_address xxxx1 tcp_outgoing_address xxxx2 tcp_outgoing_address xxxx3 tcp_outgoing_address xxxx4 tcp_outgoing_address xxxx5 http_access deny !Safe_ports http_access deny CONNECT !SSL_ports http_access allow localhost manager http_access deny manager include /etc/squid/conf.d/* http_access allow localhost http_access allow localnetA http_access allow extraIP01 http_access allow extraIP02 http_access allow extraIP03 http_access allow extraIP04 http_access allow extraIP05 http_access deny all http_port 127.0.0.1:3128 dns_v4_first on then I did export o shell aso export https_proxy=http://localhost:3128 export http_proxy=http://localhost:3128 but I still get body: '{"code":-1003,"msg":"Way too much request weight used; IP banned until 1610514306891. Please use the websocket for live updates to avoid bans."}' please check configs and let me know if there is anything to change. Thank you. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#430 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIULVKA543RGUSIIER6LRYTSZUQ4ZANCNFSM4NFXWECQ> .

[din1369]

@Jabbaxx great now i have an idea,
so you mean i need to break down my node app to several node apps which running on different ports and assign different squids for each. is that correct?

also how to have more than one squid on one server? like
http://localhost:3128
and
http://localhost:3129 ? is there any config for that?

Thank you.

[MegaManSE]

I just have one node app and scrapoxy manages/rotates the proxies if you can’t break up your app to use squid.

On Wed, Jan 13, 2021 at 02:41 din1369 ***@***.***> wrote: @Jabbaxx <https://github.com/Jabbaxx> great now i have an idea, so you mean i need to break down my node app to several node apps which running on different ports and assign different squids for each. is that correct? also how to have more than one squid on one server? like http://localhost:3128 and http://localhost:3129 ? is there any config for that? Thank you. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#430 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAUBKE47BP5QD6W65FKVK7DSZVFCLANCNFSM4NFXWECQ> .

-- Will Perone