-
Notifications
You must be signed in to change notification settings - Fork 109
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Disable express response validation in production #4810
Conversation
…DE_ENV=production
Once distributors update to new verison you should change your NODE_ENV from |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks Good.
Please run yarn generate:docs:all
in distributor-node
directory to update the API documentation
Just curious how did you figure out that this was sort of bottleneck. Was this latency information present in Opentelemetry traces.
This is express-openapi-validator
's validator function responsible for response validation. I am assuming it involves complex checks like regex matching, type checking, or nested validation for objects and arrays, which increases response times
I guess it's fine to disable validation in prod
, and delegate the validation responsibility to client applications
will do.
Actually nothing in the traces pointed at response validation being a bottleneck. I was just parsing the code and looking for something small to change that could have an impact, and from past experience I now if the response is large it could be a problem. But I tested it after disabling validation and saw a large spike in request rate. But I'm starting to wonder if this was pure chance timing..
Looking at the validation code, it looks like even if validation is enabled it seems it only does it if the content-type is JSON and skips otherwise. So It might in-fact not be having much of an affect. |
@zeeshanakram3 I set distributor-node package version to |
oof and I just changed to |
Lets merge yours first, then I'll update this one to also be part of 1.4.0 before publishing docker image. WDYT? |
A potential source of delay in processing requests for assets from storage and distributor nodes is the OpenAPI response validation that is happening because the nodes are running in "development" mode.
Most operators are likely not setting their
NODE_ENV=production
at all. But even they did, Colossus has hard-codedtrue
for response validation, and Argus only disables it ifNODE_ENV=prod
.I tested disabling response validation on my distributor node. Clearly we see a significant jump in the number requests/s the node is able to handle once the validation is disabled.
With response validation:
Without validation:
For Colossus it might also make sense to disable request validation to avoid validating large file uploads?
Distributors can immediately test this out by adding
NODE_ENV: prod
under theenvironment:
section of theirdocker-compose.yml
.Storage operators would have to pull down latest version once it is merged, and set env variable
NODE_ENV=production
If this truly makes a difference we will not feel it until all distributors and storage nodes apply this.