-
Notifications
You must be signed in to change notification settings - Fork 30
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Hough indexing from PyEBSDIndex #458
Comments
Here's an example notebook where I've tried out both PC optimization and Hough indexing in PyEBSDIndex on simulated and experimental patterns. My plan is to start a branch in this repo where we work towards supporting this functionality alongside improving PyEBSDIndex. |
This sounds like a really interesting idea! It can be very powerful if Kikuchipy can utilise a "hybrid" indexing method - using Hough indexing for a fast stage-one indexing, then re-index the pixels with low confidence using dictionary indexing. Could massively improve the index efficiency. |
Yes, that would be very cool. For that to work, we need some "logistical tools", like supporting a navigation mask (the "bad" map points) in |
PyEBSDIndex is now available on PyPI as a release candidate v0.1rc1, so we can start to depend on it for Hough indexing and PC optimization of cubic materials and NLPAR! |
In the short term, I think adding a notebook explaining how to use PyEBSDIndex with kikuchipy is best. We can then work towards making workflows simpler via convenience functions in kikuchipy and in PyEBSDIndex. |
Firstly, thank you very much for all the hard work going into the kikuchipy package! I have a question regarding the application of NLPAR from the PyEBSDIndex package on datasets saved in data formats other than the EDAX 'up1' or 'up2' formats. Is it currently possible to run this on other formats without changing code on the PyEBSDIndex side? I tried following your short tutorial in the documentation using the sample dataset from EMsoft's DITutorial ('EDAX-Ni.h5') but it has some trouble reading it. Alternatively, would you be able to make the "scan2v3.up1" data file available to run the NLPAR example on and to see how the data is saved in this format? Thanks |
@jwestraadt what kind files are you trying to process? I have been working out some bugs with NLPAR within the PyEBSD package, and while up1 and up2 (version 3) files appear to be well supported, other files are not as well tested. On my personal repo (https://github.com/drowenhorst-nrl/PyEBSDIndex), I have pushed some edits that allow for up1/2 version 1 files - but have yet to document how to use the changes. I have also tested a bit on HDF5 files. I want to make some further improvements for those, but not sure when I will get to it. Right now, I make a full copy of the HDF5 file, and put the NLPAR parameters on the end of the filename. I want to 1.) include the parameters within the hdf5 under a header directory. 2.) Make the option to save as a new datagroup/set within the HDF5 file. Unfortunately, the vendor's insistence to use different naming conventions for everything in their HDF5 files makes this a bit more difficult. Re: releasing the scan2V3.up1 ... I have to look into that. That is part of a larger dataset that I need to get permissions to release. |
@jwestraadt I should also note, I have NOT written the overall NLPAR implementation to work with an array of patterns in memory. It is assumed that the original file is the input, and that the result will be a new file. With most EBSD scans with patterns taking up 3-30GB, and knowing that a lot of people don't have 100s of GB of RAM, I made the decision to process the pattern files in chunks, and assume that there is no guarantee that even the results could be held in RAM. If you are interfacing with kikuchipy, I realize your patterns may well be in RAM, and thus this may be leading to the confusion? |
@drowenhorst-nrl I managed to get the Hough indexing to run using PyEBSDIndex on an Oxford dataset by first reading it with kikuchipy then saving it in .h5 format. Then I followed the example explained in the notebook, where the vendor was set to 'KIKUCHIPY' in the indexer. I could extract the pattern data from the .h5 file. Is it possible to follow a similar approach for NLPAR? It would be useful to have a I guess the plan is to eventually build NLPAR this into kikuchipy from the opening thread. |
@jwestraadt assuming the "patterns.h5" in the kikuchipy data repo acts like a reasonable template for the kichuchipy hdf5 format, I should have something that might work for you on my personal repo. Should be as easy as: |
Hi Johan (and Dave), thank you for the kind words! I haven't used NLPAR in PyEBSDIndex myself, so Dave has to help out there (as he has done). What I have used from PyEBSDIndex is What I'd like to test is to pass a larger-than-memory Dask array to
I would very much like that, but since PyEBSDIndex is in active development, it would require much refactoring in kikuchipy if we were to add PyEBSDIndex as a dependency. And the available developer time is very limited... I think a good approach is to try to make them work nicely together by making some convenience methods and classes where we see the need, such as the kikuchipy HDF5 file reader in PyEBSDIndex! |
Hi Dave and Håkon, Thanks, it is working quite nicely now. I could do the following:
|
That is interesting about the "use_file_locking", what size is your HDF5 file? Did you happen to also have it open in something like HDFView at the same time? |
Glad to hear that the Oxford .ebsp file reader is useful and that you got kikuchipy and PyEBSDIndex to work nicely (?) together!
If I open the >>> from h5py import File
>>> file = File("patterns.h5", mode="r+")
Traceback (most recent call last):
File "/home/hakon/miniconda3/envs/kp-dev/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3398, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-10-90063f0a65b1>", line 4, in <cell line: 4>
file = File(
File "/home/hakon/miniconda3/envs/kp-dev/lib/python3.9/site-packages/h5py/_hl/files.py", line 507, in __init__
fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
File "/home/hakon/miniconda3/envs/kp-dev/lib/python3.9/site-packages/h5py/_hl/files.py", line 222, in make_fid
fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5f.pyx", line 106, in h5py.h5f.open
BlockingIOError: [Errno 11] Unable to open file (unable to lock file, errno = 11, error message = 'Ressursen midlertidig utilgjengelig') I did not get this error when the file was not opened in HDFView, as expected. Was this what you experienced, @jwestraadt? |
Now that PyEBSDIndex is more generally available, I've added use of PyEBSDIndex to some tutorials in the docs (see links in #458 (comment)). |
#590 adds Hough indexing in kikuchipy via New issues should be raised if we decide to wrap non-local averaging (NLPAR) and a Hough transform (not indexing) from PyEBSDIndex in the future. |
After a positive experience of contributing to @drowenhorst-nrl's PyEBSDIndex package, I think it is worth working towards interfacing with the Hough indexing capabilities in that package in kikuchipy.
Before doing that though, that package must become available on PyPI (and hopefully Anaconda), so there is some work left.Note that so far only indexing of FCC and BCC materials is possible.
A rough sketch of functionality we could support:
EBSD.hough_indexing()
which takes in a lot of indexing parameters, or perhaps an indexing object/plan (like PyEBSDIndex'BandDetectPlan
) and returns aCrystalMap
, with rotations and quality metrics, and an object with other indexing metadata. Can callpyebsdindex.ebsd_index.index_pats()
on NumPy arrays as a start.EBSD.optimize_projection_center()
which via the Hough transform and Nelder-Mead optimization optimizes the PC. I guess this should return aEBSDDetector
instance.kikuchipy.pattern.hough_transform()
which returns the Hough transform of a pattern. We could also add aEBSD.hough_transform()
method that returns a new signal of Hough transformed EBSD patterns.EBSD.average_neighbour_patterns()
, and should perhaps try to improve upon this instead of using the functions in PyEBSDIndex directly.I think the focus should be on providing top-level convenience methods in kikuchipy, while more fine-grained access to the functions used in indexing by PyEBSDIndex should be accessed from that package. If this work is realized, we must take care not to duplicate efforts, and should always consider in which package stuff should go.
See also drowenhorst-nrl/PyEBSDIndex#1.Not available anymore since the repository was moved to https://github.com/USNavalResearchLaboratory/PyEBSDIndex without keeping issues.EDIT: I've added use of PyEBSDIndex in some of the tutorials in the docs:
The text was updated successfully, but these errors were encountered: