Problem: [zhashx] rehash does not increase chain limit #2174
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When rehashing in s_item_lookup(), the chain limit was not increased. Therefore, the chain limit stayed at its default value of 1. By staying at the default value of 1, a single hash collision could increase the limit of the hash. This increase could be well beyond what would be reasonable. Solution: Increase the chain limit by the growth factor after a rehash. Fixes zeromq#2173
|
Thanks! |
chu11
added a commit
to chu11/flux-core
that referenced
this pull request
Apr 15, 2021
Problem: A bug in the czmq zhashx library would incorrectly resize the hash. See: zeromq/czmq#2173 This would lead to significant performance issues as the hash would take up far more memory than it should and iteration of the hash would take an excess amount of time. A solution to this problem was fixed in: zeromq/czmq#2174 however the fix will not exist in most OS distributions for some time. Solution: We have copied in the zhashx implementation into a new convenience library libczmqcontainers. To avoid symbol collisions it has been renamed to "fzhashx" with all "zhashx" renamed to "fzhashx" respectively. While efforts were made to limit changes to the library, the following were done to avoid copying in an excess amount of code. - add fzhashx_t typedef - add #define for freen() - remove all declarations of CZMQ_EXPORT - adjust header includes
chu11
added a commit
to chu11/flux-core
that referenced
this pull request
Apr 15, 2021
Problem: A bug in the czmq zhashx library would incorrectly resize the hash. See: zeromq/czmq#2173 This would lead to significant performance issues as the hash would take up far more memory than it should and iteration of the hash would take an excess amount of time. A solution to this problem was fixed in: zeromq/czmq#2174 however the fix will not exist in most OS distributions for some time. Solution: We have copied in the zhashx implementation into a new convenience library libczmqcontainers. To avoid symbol collisions it has been renamed to "fzhashx" with all "zhashx" renamed to "fzhashx" respectively. While efforts were made to limit changes to the library, the following were done to avoid copying in an excess amount of code. - add fzhashx_t typedef - add #define for freen() - remove all declarations of CZMQ_EXPORT - adjust header includes
chu11
added a commit
to chu11/flux-core
that referenced
this pull request
Apr 16, 2021
Problem: A bug in the czmq zhashx library would incorrectly resize the hash. See: zeromq/czmq#2173 This would lead to significant performance issues as the hash would take up far more memory than it should and iteration of the hash would take an excess amount of time. A solution to this problem was fixed in: zeromq/czmq#2174 however the fix will not exist in most OS distributions for some time. Solution: We have copied in the zhashx implementation into a new convenience library libczmqcontainers. The library is copied in verbatim with only minor changes to headers and header guards. To avoid symbol collisions, add a file zhashx_map.h that will convert all internal uses of "zhashx" to "fzhashx". To avoid excess copying in from czmq, add a file czmq_internal.h that copies in macros, headers, and typdefs needed by the localized zhashx. Add a generic czmq_containers.h, that callers can include to use the internal zhashx over the shared library version.
chu11
added a commit
to chu11/flux-core
that referenced
this pull request
Apr 16, 2021
Problem: A bug in the czmq zhashx library would incorrectly resize the hash. See: zeromq/czmq#2173 This would lead to significant performance issues as the hash would take up far more memory than it should and iteration of the hash would take an excess amount of time. A solution to this problem was fixed in: zeromq/czmq#2174 however the fix will not exist in most OS distributions for some time. Solution: We have copied in the zhashx implementation into a new convenience library libczmqcontainers. The library is copied in verbatim with only minor changes to headers and header guards. To avoid symbol collisions, add a file zhashx_map.h that will convert all internal uses of "zhashx" to "fzhashx". To avoid excess copying in from czmq, add a file czmq_internal.h that copies in macros, headers, and typdefs needed by the localized zhashx. Add a generic czmq_containers.h, that callers can include to use the internal zhashx over the shared library version.
chu11
added a commit
to chu11/flux-core
that referenced
this pull request
Apr 16, 2021
Problem: A bug in the czmq zhashx library would incorrectly resize the hash. See: zeromq/czmq#2173 This would lead to significant performance issues as the hash would take up far more memory than it should and iteration of the hash would take an excess amount of time. A solution to this problem was fixed in: zeromq/czmq#2174 however the fix will not exist in most OS distributions for some time. Solution: We have copied in the zhashx implementation into a new convenience library libczmqcontainers. The library is copied in verbatim with only minor changes to headers and header guards. To avoid symbol collisions, add a file zhashx_map.h that will convert all internal uses of "zhashx" to "fzhashx". To avoid excess copying in from czmq, add a file czmq_internal.h that copies in macros, headers, and typdefs needed by the localized zhashx. Add a generic czmq_containers.h, that callers can include to use the internal zhashx over the shared library version.
chu11
added a commit
to chu11/flux-core
that referenced
this pull request
Apr 16, 2021
Problem: A bug in the czmq zhashx library would incorrectly resize the hash. See: zeromq/czmq#2173 This would lead to significant performance issues as the hash would take up far more memory than it should and iteration of the hash would take an excess amount of time. A solution to this problem was fixed in: zeromq/czmq#2174 however the fix will not exist in most OS distributions for some time. Solution: We have copied in the zhashx implementation into a new convenience library libczmqcontainers. Code was copied from the master branch at commit d4b1a1d884532e43e82b9055ceb0b84d1db5915c. The library is copied in verbatim with only minor changes to headers and header guards. To avoid excess copying in from czmq, add a file czmq_internal.h that copies in macros, headers, and typdefs needed by the localized zhashx. To avoid symbol collisions, add a file zhashx_map.h that will convert all internal uses of "zhashx" to "fzhashx". Add a generic czmq_containers.h, that callers can include to use the internal zhashx over the shared library version. All files containing copied in code maintain their czmq license (Mozilla Public License Version 2.0). All modifications to czmq files and code were isolated to those files, no code from other parts of flux-core were copied in. See: https://www.mozilla.org/en-US/MPL/2.0/combining-mpl-and-gpl/
chu11
added a commit
to chu11/flux-sched
that referenced
this pull request
Apr 17, 2021
Problem: A bug in the czmq zhashx library would incorrectly resize the hash. See: zeromq/czmq#2173 This would lead to significant performance issues as the hash would take up far more memory than it should and iteration of the hash would take an excess amount of time. A solution to this problem was fixed in: zeromq/czmq#2174 however the fix will not exist in most OS distributions for some time. Solution: We have copied in the zhashx implementation into a new convenience library libczmqcontainers. Code was copied from the master branch at commit d4b1a1d884532e43e82b9055ceb0b84d1db5915c. The library is copied in verbatim with only minor changes to headers and header guards. To avoid excess copying in from czmq, add a file czmq_internal.h that copies in macros, headers, and typdefs needed by the localized zhashx. To avoid symbol collisions, add a file zhashx_map.h that will convert all internal uses of "zhashx" to "fzhashx". Add a generic czmq_containers.h, that callers can include to use the internal zhashx over the shared library version. All files containing copied in code maintain their czmq license (Mozilla Public License Version 2.0). All modifications to czmq files and code were isolated to those files, no code from other parts of flux-sched were copied in. See: https://www.mozilla.org/en-US/MPL/2.0/combining-mpl-and-gpl/
chu11
added a commit
to chu11/flux-sched
that referenced
this pull request
Apr 17, 2021
Problem: A bug in the czmq zhashx library would incorrectly resize the hash. See: zeromq/czmq#2173 This would lead to significant performance issues as the hash would take up far more memory than it should and iteration of the hash would take an excess amount of time. A solution to this problem was fixed in: zeromq/czmq#2174 however the fix will not exist in most OS distributions for some time. Solution: We have copied in the zhashx implementation into a new convenience library libczmqcontainers. Code was copied from the master branch at commit d4b1a1d884532e43e82b9055ceb0b84d1db5915c. The library is copied in verbatim with only minor changes to headers and header guards. To avoid excess copying in from czmq, add a file czmq_internal.h that copies in macros, headers, and typdefs needed by the localized zhashx. To avoid symbol collisions, add a file zhashx_map.h that will convert all internal uses of "zhashx" to "fzhashx". Add a generic czmq_containers.h, that callers can include to use the internal zhashx over the shared library version. All files containing copied in code maintain their czmq license (Mozilla Public License Version 2.0). All modifications to czmq files and code were isolated to those files, no code from other parts of flux-sched were copied in. See: https://www.mozilla.org/en-US/MPL/2.0/combining-mpl-and-gpl/
chu11
added a commit
to chu11/flux-sched
that referenced
this pull request
Apr 18, 2021
Problem: A bug in the czmq zhashx library would incorrectly resize the hash. See: zeromq/czmq#2173 This would lead to significant performance issues as the hash would take up far more memory than it should and iteration of the hash would take an excess amount of time. A solution to this problem was fixed in: zeromq/czmq#2174 however the fix will not exist in most OS distributions for some time. Solution: We have copied in the zhashx implementation into a new convenience library libczmqcontainers. Code was copied from the master branch at commit d4b1a1d884532e43e82b9055ceb0b84d1db5915c. The library is copied in verbatim with only minor changes to headers and header guards. To avoid excess copying in from czmq, add a file czmq_internal.h that copies in macros, headers, and typdefs needed by the localized zhashx. To avoid symbol collisions, add a file zhashx_map.h that will convert all internal uses of "zhashx" to "fzhashx". Add a generic czmq_containers.h, that callers can include to use the internal zhashx over the shared library version. All files containing copied in code maintain their czmq license (Mozilla Public License Version 2.0). All modifications to czmq files and code were isolated to those files, no code from other parts of flux-sched were copied in. See: https://www.mozilla.org/en-US/MPL/2.0/combining-mpl-and-gpl/
milroy
pushed a commit
to milroy/flux-sched
that referenced
this pull request
May 25, 2021
Problem: A bug in the czmq zhashx library would incorrectly resize the hash. See: zeromq/czmq#2173 This would lead to significant performance issues as the hash would take up far more memory than it should and iteration of the hash would take an excess amount of time. A solution to this problem was fixed in: zeromq/czmq#2174 however the fix will not exist in most OS distributions for some time. Solution: We have copied in the zhashx implementation into a new convenience library libczmqcontainers. Code was copied from the master branch at commit d4b1a1d884532e43e82b9055ceb0b84d1db5915c. The library is copied in verbatim with only minor changes to headers and header guards. To avoid excess copying in from czmq, add a file czmq_internal.h that copies in macros, headers, and typdefs needed by the localized zhashx. To avoid symbol collisions, add a file zhashx_map.h that will convert all internal uses of "zhashx" to "fzhashx". Add a generic czmq_containers.h, that callers can include to use the internal zhashx over the shared library version. All files containing copied in code maintain their czmq license (Mozilla Public License Version 2.0). All modifications to czmq files and code were isolated to those files, no code from other parts of flux-sched were copied in. See: https://www.mozilla.org/en-US/MPL/2.0/combining-mpl-and-gpl/
milroy
pushed a commit
to milroy/flux-sched
that referenced
this pull request
May 25, 2021
Problem: A bug in the czmq zhashx library would incorrectly resize the hash. See: zeromq/czmq#2173 This would lead to significant performance issues as the hash would take up far more memory than it should and iteration of the hash would take an excess amount of time. A solution to this problem was fixed in: zeromq/czmq#2174 however the fix will not exist in most OS distributions for some time. Solution: We have copied in the zhashx implementation into a new convenience library libczmqcontainers. Code was copied from the master branch at commit d4b1a1d884532e43e82b9055ceb0b84d1db5915c. The library is copied in verbatim with only minor changes to headers and header guards. To avoid excess copying in from czmq, add a file czmq_internal.h that copies in macros, headers, and typdefs needed by the localized zhashx. To avoid symbol collisions, add a file zhashx_map.h that will convert all internal uses of "zhashx" to "fzhashx". Add a generic czmq_containers.h, that callers can include to use the internal zhashx over the shared library version. All files containing copied in code maintain their czmq license (Mozilla Public License Version 2.0). All modifications to czmq files and code were isolated to those files, no code from other parts of flux-sched were copied in. See: https://www.mozilla.org/en-US/MPL/2.0/combining-mpl-and-gpl/
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When rehashing in s_item_lookup(), the chain limit was not increased.
Therefore, the chain limit stayed at its default value of 1. By staying
at the default value of 1, a single hash collision could increase the
limit of the hash. This increase could be well beyond what would be
reasonable.
Solution: Increase the chain limit by the growth factor after a rehash.
Fixes #2173