Cosym target: faster numpy operations #1639
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is much faster and give ~6-fold speedup to target.compute_functional() (based on Rij matrix of size 400x400).
Remove the need for one tmp variable, pre-declare another and re-use by passing this as the `out=` parameter to np operations.
Remove need for explicit loop over number of dimensions
Avoid doubly nested loop over number of dimensions.
|
@dwpaley it would be interesting to hear what performance impact this has on your use case. |
Codecov Report
@@ Coverage Diff @@
## main #1639 +/- ##
==========================================
- Coverage 66.63% 66.62% -0.02%
==========================================
Files 616 616
Lines 68949 68926 -23
Branches 9601 9593 -8
==========================================
- Hits 45943 45920 -23
Misses 21070 21070
Partials 1936 1936 |
benjaminhwilliams
requested changes
Mar 30, 2021
There was a problem hiding this comment.
I haven't seen the algebra, so I can't check the consistency of the code with the desired arithmetic, but I presume that's not at question here. Changes look good and improve clarity, with a nice performance improvement. I have one small suggestion (affects two places). I haven't tested my suggestion though, so YMMV. Otherwise, LGTM.
Co-authored-by: Ben Williams <benjaminhwilliams@users.noreply.github.com>
benjaminhwilliams
approved these changes
Mar 30, 2021
benjaminhwilliams
approved these changes
Mar 30, 2021
algorithms/symmetry/cosym/target.py
Outdated
| for i in range(self.dim): | ||
| grad[i * NN : (i + 1) * NN] = np.matmul(wrij_matrix, coords[i]) | ||
| grad[i] = np.matmul(wrij_matrix, coords[i]) |
There was a problem hiding this comment.
matmul can also take stacks of matrices and would interpret it as such if coords dimensions is > 2. I think
grad = np.matmul(wrij_matrix, coords) should also work
Maybe for future considerations
ndevenish
pushed a commit
that referenced
this pull request
Mar 31, 2021
Significant speedup of computation of cosym target functional, gradients and curvatures. * Use np.square(x) instead of np.power(x, 2) * Avoid explicit loops over number of dimensions * Functional now ~10x quicker * Gradients ~20x quicker * Curvatures ~4x quicker
DiamondLightSource-build-server
added a commit
that referenced
this pull request
Mar 31, 2021
Features -------- - ``dials.cosym``: Significantly faster via improved computation of functional, gradients and curvatures (#1639) - ``dials.integrate``: Added parameter ``valid_foreground_threshold=``, to require a minimum fraction of valid pixels before profile fitting is attempted (#1640) Bugfixes -------- - ``dials.cosym``: Cache cases where Rij is undefined, rather than recalculating each time. This can have significant performance benefits when handling large numbers of sparse data sets. (#1634) - ``dials.cosym``: Fix factor of 2 error when calculating target weights (#1635) - ``dials.cosym``: Fix broken ``engine=scipy`` option (#1636) - ``dials.integrate``: Reject reflections with a high number of invalid pixels, which were being integrated since 3.4.0. This restores better merging statistics, and prevents many reflections being incorrect profiled as zero-intensity. (#1640)
Merged
DiamondLightSource-build-server
added a commit
that referenced
this pull request
Apr 1, 2021
Features -------- - ``dials.cosym``: Significantly faster via improved computation of functional, gradients and curvatures (#1639) - ``dials.integrate``: Added parameter ``valid_foreground_threshold=``, to require a minimum fraction of valid pixels before profile fitting is attempted (#1640) Bugfixes -------- - ``dials.cosym``: Cache cases where Rij is undefined, rather than recalculating each time. This can have significant performance benefits when handling large numbers of sparse data sets. (#1634) - ``dials.cosym``: Fix factor of 2 error when calculating target weights (#1635) - ``dials.cosym``: Fix broken ``engine=scipy`` option (#1636) - ``dials.integrate``: Reject reflections with a high number of invalid pixels, which were being integrated since 3.4.0. This restores better merging statistics, and prevents many reflections being incorrect profiled as zero-intensity. (#1640)
DiamondLightSource-build-server
added a commit
that referenced
this pull request
Apr 1, 2021
Features -------- - ``dials.cosym``: Significantly faster via improved computation of functional, gradients and curvatures (#1639) - ``dials.integrate``: Added parameter ``valid_foreground_threshold=``, to require a minimum fraction of valid pixels before profile fitting is attempted (#1640) Bugfixes -------- - ``dials.cosym``: Cache cases where Rij is undefined, rather than recalculating each time. This can have significant performance benefits when handling large numbers of sparse data sets. (#1634) - ``dials.cosym``: Fix factor of 2 error when calculating target weights (#1635) - ``dials.cosym``: Fix broken ``engine=scipy`` option (#1636) - ``dials.integrate``: Reject reflections with a high number of invalid pixels, which were being integrated since 3.4.0. This restores better merging statistics, and prevents many reflections being incorrect profiled as zero-intensity. (#1640) - Fix rare crash in symmetry calculations when no resolution limit could be calculated (#1641)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Significant speedup of computation of cosym target functional, gradients and curvatures.
Use
np.square(x)instead ofnp.power(x, 2)Avoid explicit loops over number of dimensions
Functional now ~10x quicker
Gradients ~20x quicker
Curvatures ~4x quicker
For a real dials.cosym example, 44 datasets in P3121, this change cut about 25s off the total runtime: