receive: allow custom connection pooling

[blockloop]

• I added CHANGELOG entry for this change.
• Change is not relevant to the end user.

Changes

Receive: enable custom connection pool sizing. With the default configuration machines with very high load (over 500 requests per second) will cycle TCP connections too frequently and run out of available ports. See more here

Verification

Running a custom build in production. Also see #1955

[squat]

👍 awesome stuff

[squat]

👍 awesome stuff

[blockloop]

I think the build needs to be kicked @bwplotka
Also, I addressed your change requests

[brancz]

Somewhat related, I was thinking we should switch the internal communications to use gRPC, which could eventually evolve to also be the externally accepting write request API. I’m mentioning this as that would probably make this obsolete. What do you think?

[blockloop]

I agree. I think this should be gRPC streams all the way through. I have a branch on my fork with a lot of that work complete. https://github.com/blockloop/thanos/tree/bjones/receive/use-grpc

[squat]

Happy to review a PR switching the internal forwarding to gRPC. I say switching rather than adding as there is no use for two internal forwarding mechanisms. El El mié, ene. 8, 2020 a las 19:07, Brett <notifications@github.com> escribió:

…

I agree. I think this should be gRPC streams all the way through. I have a branch on my fork with a lot of that work complete. https://github.com/blockloop/thanos/tree/bjones/grpc-remote-write — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1966?email_source=notifications&email_token=AE4JAPYZ54SBK2MTLEAZ253Q4YI6JA5CNFSM4KELD3O2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEINOORA#issuecomment-572188484>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE4JAP62GE2RXXGVR5PEF4LQ4YI6JANCNFSM4KELD3OQ> .

[blockloop]

@squat, I think that makes sense. Right now since forwarding and receiving are the same code that would require a bit of a change. I can change gears a bit and start on splitting away forwarding from receiving HTTP remote writes. That might be the better option. Ideally the gRPC endpoint would also accept remote writes externally as well.

[squat]

Let me know if you want to pair. I think it should be quite straight forward to essentially keep the same forward func and simply have two handlers (one gRPC and one HTTP) that both call the same forwarding func. El El mié, ene. 8, 2020 a las 21:30, Brett <notifications@github.com> escribió:

…

@squat <https://github.com/squat>, I think that makes sense. Right now since forwarding and receiving are the same code that would require a bit of a change. I can change gears a bit and start on splitting away forwarding from receiving HTTP remote writes. That might be the better option. Ideally the gRPC endpoint would also accept remote writes externally as well. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1966?email_source=notifications&email_token=AE4JAP7UI5TOYHBCWQ7ONSDQ4YZXNA5CNFSM4KELD3O2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIN342A#issuecomment-572243560>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE4JAPYCDZRBE6HE3DAHDQ3Q4YZXNANCNFSM4KELD3OQ> .

[blockloop]

Yeah I think a RemoteWrite method needs to be added to store.TSDBStore though, right?

[blockloop]

Maybe we should shift the conversation to #1710 and close this PR?

[squat]

Yeah it’ll have to be a new service that has the store methods as well as the RemoteWrite method

[blockloop]

Closing this in favor of #1970