Utility functions for iterative algorithms #60

uellue · 2021-09-23T15:48:48Z

Fast image transforms using a sparse matrix product
Coordinate transforms to match detector data with forward model
Calculate stacks of shifted and rolled products of a smaller illumination within a larger object, including sub-pixel shifting
Aggregate shifted and rolled updates within a larger object

The functions can perform FFT shifts on the fly to reduce memory transfers. Both CPU and CUDA are supported. They are tested to conform precisely to the conventions of libertem.api.create_com_analysis() and also SSB_UDF.

CC @bangunarya @Sniper2k that's the stuff I have mentioned recently. If you'd like to try it, let me know! :-)

Contributor Checklist:

I have added myself to the creators.json file
I have added a changelog entry for my contribution
I have added/updated documentation for all user-facing changes
I have added/updated test cases

Intended for extracting data in such a way that it matches the reconstruction grid. This is a performance-critical and error-prone step in iterative ptychography algorithms.

* Image and coordinate transformations * Shifted, cropped and rolled products and aggregation

src/ptychography40/reconstruction/common.py

Co-authored-by: Dieter Weber <d.weber@fz-juelich.de>

SimeonEhrig · 2021-09-29T11:50:42Z

@uellue Is there are a rendered version of the sphinx doc available? In the alpaka-group, we use readthedoc. There is a CI job, which build the sphinx doc and provides an link to the result.

uellue · 2021-09-29T12:28:32Z

@uellue Is there are a rendered version of the sphinx doc available? In the alpaka-group, we use readthedoc. There is a CI job, which build the sphinx doc and provides an link to the result.

At the moment the rendered documentation is only available after merging or by running tox -e docs locally. @sk1p, having the rendered docs of a PR available could help with reviewing, since sometimes it builds cleanly but looks terrible. I've reviewed the rendered docs locally up to now, which is a bit tedious.

SimeonEhrig · 2021-09-29T12:32:05Z

At the moment the rendered documentation is only available after merging or by running tox -e docs locally. @sk1p, having the rendered docs of a PR available could help with reviewing, since sometimes it builds cleanly but looks terrible. I've reviewed the rendered docs locally up to now, which is a bit tedious.

Thanks. I was not sure, if the CI job provides an artifact of the build. But building locally is also fine.

* Shift real and imaginary parts separately to support complex illumination * Iterate only over shape of updates, not the whole object, to improve scaling behavior with large objects and small updates. Use atomic add to avoid conflict between threads and blocks writing to the same memory.

sk1p

I've added some comments and typo fixes. Maybe we can also make some diagrams for describing the coordinate systems involved, and how the transformations work?

Thanks!

src/ptychography40/reconstruction/common.py

docs/source/algorithms.rst

Co-authored-by: Alexander Clausen <alex@gc-web.de>

codecov · 2021-10-25T15:03:59Z

Codecov Report

Merging #60 (ef0c3e6) into master (fec553e) will decrease coverage by 4.15%.
The diff coverage is 53.98%.

@@            Coverage Diff             @@
##           master      #60      +/-   ##
==========================================
- Coverage   64.08%   59.92%   -4.16%     
==========================================
  Files           8        8              
  Lines         323      549     +226     
  Branches       38       66      +28     
==========================================
+ Hits          207      329     +122     
- Misses        104      206     +102     
- Partials       12       14       +2

Impacted Files	Coverage Δ
src/ptychography40/reconstruction/common.py	`54.54% <53.98%> (-2.13%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fec553e...ef0c3e6. Read the comment docs.

Co-authored-by: Alexander Clausen <alex@gc-web.de>

sk1p · 2021-10-27T13:16:52Z

It looks like the current mac os x image doesn't include Python 3.6 anymore - we may need to switch to an older one (at least for 3.6). See here: https://github.com/actions/virtual-environments/blob/main/images/macos/toolsets/toolset-10.14.json https://github.com/actions/virtual-environments/blob/main/images/macos/toolsets/toolset-11.json

@sk1p

Thx @sk1p! * Example notebook with more detail * Coordinate transform is always a factory function, no need for functools

* Also turn (i)fftshift_coords() into higher-order functions * Clarify and tweak example * Docs updates for clarity

uellue · 2021-10-29T09:01:55Z

It looks like the current mac os x image doesn't include Python 3.6 anymore - we may need to switch to an older one (at least for 3.6). See here: https://github.com/actions/virtual-environments/blob/main/images/macos/toolsets/toolset-10.14.json https://github.com/actions/virtual-environments/blob/main/images/macos/toolsets/toolset-11.json

Should I do sth about that in this PR? Can you give me a pointer?

sk1p · 2021-10-29T09:39:41Z

Should I do sth about that in this PR? Can you give me a pointer?

You could try to set the vmImage in the mac os x matrix to macos-10.14 for the Python36 entry here:

ptychography/azure-pipelines.yml

Lines 192 to 202 in 18114c5

    
           strategy: 
        
             matrix: 
        
               Python36: 
        
                 python.version: '3.6' 
        
                 TOXENV: 'py36' 
        
               Python37: 
        
                 python.version: '3.7' 
        
                 TOXENV: 'py37' 
        
               Python38: 
        
                 python.version: '3.8' 
        
                 TOXENV: 'py38'

Should be possible to just specify vmImage as sibling of TOXENV and python.version, like this example.

sk1p · 2021-10-29T09:44:21Z

See also LiberTEM/LiberTEM#1142

@sk1p

Thx @sk1p! :-)

sk1p · 2021-10-29T09:49:52Z

Hm, that didn't work - still showing "Image: macOS-latest"...

Following LiberTEM/LiberTEM#1142

uellue · 2021-11-12T08:53:36Z

@sk1p is the notebook that shows the transforms understandable? Do we need more documentation, are there other things to address? It would be nice to merge this some time next week to make it easier to use in downstream code. :-)

sk1p · 2021-11-12T12:15:54Z

is the notebook that shows the transforms understandable?

Yes, thanks!

Do we need more documentation, are there other things to address? It would be nice to merge this some time next week to make it easier to use in downstream code. :-)

I think documentation can always be improved, but let's merge this now (perfect is the enemy of good, and all that...)

uellue added 2 commits September 16, 2021 19:19

Construct sparse matrices to transform images

ab1af0e

Intended for extracting data in such a way that it matches the reconstruction grid. This is a performance-critical and error-prone step in iterative ptychography algorithms.

Utility functions for iterative algorithms

b5700d3

* Image and coordinate transformations * Shifted, cropped and rolled products and aggregation

uellue added the enhancement New feature or request label Sep 23, 2021

uellue added this to the 0.2 milestone Sep 23, 2021

creators.json for utility functions

68e7014

uellue requested review from sk1p, bangunarya, SimeonEhrig and Sniper2k September 23, 2021 16:31

uellue commented Sep 27, 2021

View reviewed changes

src/ptychography40/reconstruction/common.py Outdated Show resolved Hide resolved

Update src/ptychography40/reconstruction/common.py

bf1477c

Co-authored-by: Dieter Weber <d.weber@fz-juelich.de>

SimeonEhrig approved these changes Sep 29, 2021

View reviewed changes

uellue added 3 commits October 7, 2021 09:40

Work around missing complex atomic add

6fadb8b

Not enough blocks on RTX 3090; atomic add allows more concurrency

3c4eb22

uellue mentioned this pull request Oct 12, 2021

Software for ptychography reconstruction and other quantitative interpretation of diffraction patterns FAIRmat-NFDI/AreaB-software-tools#6

Open

sk1p reviewed Oct 20, 2021

View reviewed changes

uellue and others added 2 commits October 25, 2021 16:59

Update src/ptychography40/reconstruction/common.py

9f7292a

Co-authored-by: Alexander Clausen <alex@gc-web.de>

Update docs/source/algorithms.rst

bfcb5b6

Co-authored-by: Alexander Clausen <alex@gc-web.de>

Update src/ptychography40/reconstruction/common.py

16dc096

Co-authored-by: Alexander Clausen <alex@gc-web.de>

uellue added 2 commits October 28, 2021 17:26

WIP feedback

4b54fa5

Thx @sk1p! * Example notebook with more detail * Coordinate transform is always a factory function, no need for functools

Finish review @sk1p

ac04126

* Also turn (i)fftshift_coords() into higher-order functions * Clarify and tweak example * Docs updates for clarity

uellue requested a review from sk1p October 29, 2021 09:02

uellue mentioned this pull request Oct 29, 2021

Numba code coverage #62

Closed

Older OS X for Py 3.6

887efdd

Thx @sk1p! :-)

uellue and others added 2 commits October 29, 2021 11:53

Tweak image name

276b988

Following LiberTEM/LiberTEM#1142

use $(imageName) for pool.vmImage

ef0c3e6

sk1p approved these changes Nov 12, 2021

View reviewed changes

sk1p merged commit db04eee into Ptychography-4-0:master Nov 12, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Utility functions for iterative algorithms #60

Utility functions for iterative algorithms #60

uellue commented Sep 23, 2021 •

edited

SimeonEhrig commented Sep 29, 2021

uellue commented Sep 29, 2021

SimeonEhrig commented Sep 29, 2021

sk1p left a comment

codecov bot commented Oct 25, 2021 •

edited

sk1p commented Oct 27, 2021

uellue commented Oct 29, 2021 •

edited

sk1p commented Oct 29, 2021

sk1p commented Oct 29, 2021

sk1p commented Oct 29, 2021

uellue commented Nov 12, 2021

sk1p commented Nov 12, 2021

Utility functions for iterative algorithms #60

Utility functions for iterative algorithms #60

Conversation

uellue commented Sep 23, 2021 • edited

Contributor Checklist:

SimeonEhrig commented Sep 29, 2021

uellue commented Sep 29, 2021

SimeonEhrig commented Sep 29, 2021

sk1p left a comment

Choose a reason for hiding this comment

codecov bot commented Oct 25, 2021 • edited

Codecov Report

sk1p commented Oct 27, 2021

uellue commented Oct 29, 2021 • edited

sk1p commented Oct 29, 2021

sk1p commented Oct 29, 2021

sk1p commented Oct 29, 2021

uellue commented Nov 12, 2021

sk1p commented Nov 12, 2021

uellue commented Sep 23, 2021 •

edited

codecov bot commented Oct 25, 2021 •

edited

uellue commented Oct 29, 2021 •

edited