Use shorter identifier for ConcatenatedModules #5997
Conversation
See #5992 (comment) for rationale. Fixes #5992
| case "concatenated": | ||
| return info.module.identifier(); | ||
| } | ||
| }).filter(Boolean).join(" "); |
There was a problem hiding this comment.
Instead of mapping, filtering, and joining, I wonder if this would be faster and use less memory if you call hash.update while iterating in a forEach:
const hash = crypto.createHash("md5");
this._orderedConcatenationList.forEach(info => {
if (info.type === "concatenated") {
hash.update(info.module.identifier() + " ");
}
});
return this.rootModule.identifier() + " " + hash.digest("hex");There was a problem hiding this comment.
Not sure, but the difference is probably negligible. Profiling should tell us either way.
If we want raw speed, md4 might be worth considering as well. Not sure about collisions though.
There was a problem hiding this comment.
Oh, I've also had good success using string-hash in client code, but that isn't set up to accept input in chunks like crypto stuff. https://github.com/darkskyapp/string-hash
There was a problem hiding this comment.
That sounds good to me. I think in that case, this PR should stick with md5 and then can be upgraded with whatever you land on after that testing is complete.
I'd love to try this out at Airbnb ASAP--is there any chance we can get this in and get a new webpack 3 release out?
There was a problem hiding this comment.
My only reasoning for picking this algorithm was consistency really. It is what NormalModule is using. Happy to change it to whatever.
There was a problem hiding this comment.
Cleaned out the comments that really have no bearing on this particular PR.
@lencioni - I do believe that's the plan though webpack/webpack isn't my domain. I'm going to work on the hash testing now. Assuming the results are positive, i'll tag you in that PR as well given it's probably going to be applicable to airbnb's perf issues as well.
There was a problem hiding this comment.
@filipesilva - Leave it the way it is, we kind of went off on a tangent that is likely to lead to more perf improvements but needs to be done across the entire project.
|
Thank you for your pull request! The most important CI builds succeeded, we’ll review the pull request soon. |
|
Thanks |
This was implemented in webpack#5997 to fix some memory issues, but I think we can make this a little more efficient. By using a single forEach instead of a map, filter, and join, we avoid unnecessary array creations and iterations.
This was implemented in webpack#5997 to fix some memory issues, but I think we can make this a little more efficient. By using a single forEach instead of a map, filter, and join, we avoid unnecessary array creations and iterations.
|
I just tried out v3.9.1 at Airbnb, and the memory problems were were seeing seem to be resolved now! Thanks so much for this! |
webpack@~3.11.0 includes a fix for an out-of-memory issue; see: webpack/webpack#5992 webpack/webpack#5997
* Upgrade webpack to ~3.11.0 webpack@~3.11.0 includes a fix for an out-of-memory issue; see: webpack/webpack#5992 webpack/webpack#5997 * rush change
What kind of change does this PR introduce?
Memory usage optimization.
Did you add tests for your changes?
I couldn't find a test for module names, it doesn't seem like it is tested currently.
If relevant, link to documentation update:
Summary
See #5992 (comment) for rationale.
Fixes #5992
Does this PR introduce a breaking change?
No.
Other information