Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Log the image template #2278

Merged
merged 53 commits into from
Mar 16, 2023
Merged

Log the image template #2278

merged 53 commits into from
Mar 16, 2023

Conversation

dagewa
Copy link
Member

@dagewa dagewa commented Nov 11, 2022

WIP for #1662

This works for scans, but not for stills. Could somebody who works with stills please add in the appropriate code path? I don't know how to get a template out of an ImageSet and anyway I don't know how you want to report the possibly cherry-picked image numbers.

@dagewa
Copy link
Member Author

dagewa commented Nov 11, 2022

Tagging @graeme-winter, @phyy-nx and @rjgildea as those who commented on #1662

@codecov
Copy link

codecov bot commented Nov 11, 2022

Codecov Report

Merging #2278 (74bc65c) into main (cb7ac03) will decrease coverage by 0.07%.
The diff coverage is 100.00%.

❗ Current head 74bc65c differs from pull request most recent head ef514f6. Consider uploading reports for the commit ef514f6 to get more accurate results

@@            Coverage Diff             @@
##             main    #2278      +/-   ##
==========================================
- Coverage   80.61%   80.54%   -0.07%     
==========================================
  Files         587      587              
  Lines       67306    67320      +14     
  Branches     8995     8999       +4     
==========================================
- Hits        54258    54223      -35     
- Misses      10972    11018      +46     
- Partials     2076     2079       +3     

@dagewa
Copy link
Member Author

dagewa commented Feb 20, 2023

Tempted to merge this, because it is an improvement for users processing scans and doesn't change the behaviour for users processing stills. Any thoughts?

@dagewa dagewa marked this pull request as ready for review February 20, 2023 09:56
dagewa and others added 19 commits February 20, 2023 10:04
Fixes #2279
Co-authored-by: James Beilsten-Edmands <30625594+jbeilstenedmands@users.noreply.github.com>
As of wxPython 4.2.0 passing a float results in an error:

Timer.Start(): argument 1 has unexpected type 'float'
* Improve comments. "Incident beam" is ambiguous as it might mean
incident on the sample, or on the detector.

* Add link to the relevant XDS paper

* Factor is less ambiguous than fraction

* Correct file name
NoQA for typing imports
* Pin boost to 1.74

conda currently picks 1.78, which requires c++14.  This is failing for dxtbx/dials.
dev.dials.napari_rlv is an alternative reciprocal lattice viewer based on napari.
Don't count zero as negative. Fixes #2084
Co-authored-by: Aaron Brewster <asbrewster@lbl.gov>
Co-authored-by: Iris Young <idyoung@gmail.com>
…0` (#2298)

Allow `reflection_table.remove_on_experiment_identifiers` when `len==0`
Add new feature to dials.stills_process: reflection_subsampling. If reflection_subsampling.enable=True, and an image fails to index, the program will randomly subsample the reflections and try again. It keeps trying, randomly subsampling in smaller amounts until a threshold is reached or the image successfully indexes.

The default is descending from 100% to 50% of the strong spots by 2% steps, randomly subsampling from the original pool of strong reflections at each step.

We have observed nearly doubling the number of indexed images for some samples using this approach, and we verified that the additional lattices are high quality (ie we are not adding merely weak lattices that failed to index).

* Add test for reflection_subsampling

The gain of the SACLA MPCCD detector is normally 10.  The new test wrong_gain changes it to 0.5 and produce more spots, which causes the 3rd image to fail to index. This is rescued by reflection_subsampling.

Co-authored-by: Daniel Paley dwpaley@lbl.gov
Co-authored-by: Asmit Bhowmick abhowmick@lbl.gov
jbeilstenedmands and others added 24 commits February 20, 2023 11:09
… during scaling (#2312)

Only triggered if a dataset being removed is near the end of the experimentlist
When selecting the odd experiments in a large reflection table from a
serial dataset, this improves the time for the select statement from
~10 min to ~1 second.
This was counting number of clusters instead of size of each cluster.
Co-authored-by: Richard Gildea <rjgildea@users.noreply.github.com>
Previously UnboundLocalError was raised when
absorption_correction.apply=True, but absorption_correction.algorithm
was unspecified.

Now, whenever algorithm= is expected but unspecified, a ValueError is
raised, unless user specifically requests algorithm=other.
These functions expect ints

Co-authored-by: James Beilsten-Edmands <30625594+jbeilstenedmands@users.noreply.github.com>
Prevent dials.combine_experiments from raising TypeError in cases
where clustering.use=True but the default clustering.max_clusters=None.

Instead, interpret max_clusters=None as requesting all of the clusters.
Using C++14 by default exposes dials to failures on windows because of
the standard library overrides in cctbx, so remove those when building
dials libraries.

Using C++14 means that we can use Boost::Geometry in boost 1.78, which
dropped support for older versions. This means that we can build against
the January 2023 release of cctbx, which requires boost 1.78.

Additionally, dxtbx directly links to zlib but this was only ever pulled
into the conda environments implicitly, but on some configurations of
building dials on windows this is no longer the case - so make the
dependency explicit here.

Fixes #2291.
subprocess.Popen docs recommend passing a fully-qualified path to the
executable, as resolving the path of the executable is platform
dependent.
https://docs.python.org/3/library/subprocess.html#subprocess.Popen

This was causing test failures in some windows configurations.
We expect a C++11 compiler, so we may as well use standard C++11
features where available instead of legacy boost equivalents.

- Use std::atomic instead of boost::atomic
- Use std::shared_ptr instead of boost::shared_ptr
- Use std::make_shared instead of boost::make_shared
- Use standard <random> instead of boost equivalent
- Use std::dynamic_pointer_cast
- Remove unused boost includes
    towncrier --name=DIALS --version='3.13.0'
Historically this has been allowed to lag. We should try to move towards
a more consistently updated value for future releases.
This gets the correct allocated core count on Slurm.
libtbx.introspection.number_of_processors does not.
If calling phil.extract() raises an exception (for example
because the provided value is of the wrong type or out of
range) then catch the error and call parser.error() which
results in a call to sys.exit().
* Move a block from run to run_dials_refine

* Ensure sys.exit only appears under run, not under run_dials_refine

* forgot the news

* Rename newsfragments/xxx.bugfix to newsfragments/2325.bugfix

---------

Co-authored-by: DiamondLightSource-build-server <DiamondLightSource-build-server@users.noreply.github.com>
@dagewa
Copy link
Member Author

dagewa commented Feb 20, 2023

Sigh. Implicit rebase when I did a git pull has left this all messy :-(

@phyy-nx
Copy link
Member

phyy-nx commented Feb 21, 2023

Ok with me. Sorry I haven't had a chance to work on it.

@dagewa
Copy link
Member Author

dagewa commented Mar 2, 2023

Just to note, this is also relevant for #2072, though that request is also to put this information in spot-finding and integration logs, so this does not strictly close it.

@jbeilstenedmands I think you said you might look at this for stills? If not soon, I'd be keen to merge this.

@dagewa dagewa merged commit 156f0fd into main Mar 16, 2023
@dagewa dagewa deleted the log-image-template branch March 16, 2023 10:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet