Open pixelated STEM framework
Clone or download
sk1p is_local is now set automatically iif you use .connect(...)
 + removed isLocal checkbox from client UI and message
Latest commit 853a6f9 Dec 13, 2018
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
benchmarks hdfs/bench_buffering: switch to default port 8020 May 8, 2018
client is_local is now set automatically iif you use .connect(...) Dec 13, 2018
corporatedesign Include logo, acknowledgments and DOI in GUI Nov 20, 2018
docs Dimensionality refactoring (#195) Dec 12, 2018
examples [skip ci] Move to prototypes Dec 13, 2018
fileformat remove fileformat related files Aug 1, 2018
packaging build and upload wheel and sdist package, refs #142 Nov 27, 2018
prototypes do filesystem operations via the executor, refs #205 Dec 13, 2018
scripts Fixed Windows permission errors Nov 5, 2018
src/libertem is_local is now set automatically iif you use .connect(...) Dec 13, 2018
tests tests: partition shape can vary by number of cores Dec 13, 2018
.gitattributes add gitattributes to mark ts build output as binary Oct 9, 2018
.gitignore [skip ci] Ignore .vscode directory with folder-specific settings Dec 10, 2018
.mailmap fix mailmap syntax [skip ci] Dec 12, 2018
.travis.yml Correct way to set global environment for Travis Dec 6, 2018
.zenodo.json Some more JSON issues Nov 5, 2018
LICENSE add LICENSE(s), reorganize packages/modules Jul 11, 2018
MANIFEST.in include README.rst in sdist, add MANIFEST to gitignore Oct 25, 2018
README.rst Zenodo badge updated and included in README; favicon Nov 7, 2018
appveyor.yml Include start of real cluster in tests Dec 6, 2018
pyproject.toml Workaround for a pip bug Dec 12, 2018
requirements.txt Portable way to determine file owner Nov 28, 2018
setup.cfg Dimensionality refactoring (#195) Dec 12, 2018
setup.py Portable way to determine file owner Nov 28, 2018
tox.ini Dimensionality refactoring (#195) Dec 12, 2018

README.rst

gitter travis appveyor zenodo

LiberTEM is an open source platform for high-throughput distributed processing of pixelated scanning transmission electron microscopy (STEM) data.

It is designed for high throughput and scalability on PCs, single server nodes, clusters and cloud services. On clusters it can use the Hadoop file system with fast distributed local storage on high-performance SSDs. That way it achieves very high collective IO performance on a compact and cost-efficient system built from stock components. With cached file system reads it can reach a throughput of up to 14 GB/s per processing node with a quad-core CPU.

LiberTEM is supported on Linux, Mac OS X and Windows. Other platforms that allow installation of Python 3 and the required packages will likely work as well. The GUI is running in a web browser.

LiberTEM currently opens most file formats used for pixelated STEM:

  • Raw binary files, for example for the Thermo Fisher EMPAD detector
  • Quantum Detectors MIB format (currently alpha, more testing and sample files highly appreciated)
  • Nanomegas .blo block files
  • Gatan K2IS raw format (currently beta)
  • HDF5-based formats such as Hyperspy files, NeXus and EMD
  • Please contact us if you are interested in support for an additional format!

The 0.1 release implements anything that can be done by applying masks to each detector frame, for example the numerous virtual detector methods (virtual bright field, virtual HAADF, ...) or center of mass.

The GUI of the 0.1 release allows interactive analysis and data exploration with basic virtual detectors, center of mass and display of individual detector frames.

The Python API can be used for more complex operations with arbitrary masks and other features like data export. There are example Jupyter notebooks available in the examples directory. If you are having trouble running the examples, please let us know, either by filing an issue or by joining our Gitter chat.

LiberTEM is suitable as a high-performance processing backend for other applications, including live data streams. Contact us if you are interested!

Deployment as a single-node system for a local user is thoroughly tested and can be considered stable in the 0.1 release. Deployment on a cluster is experimental and still requires some additional work, see Issue #105.

LiberTEM is evolving rapidly and prioritizes features following user demand and contributions. In the future we'd like to implement live acquisition, and more analysis methods for all applications of pixelated STEM. If you like to influence the direction this project is taking, or if you'd like to contribute, please join our gitter chat, our development mailing list, and our general mailing list.

Installation

The short version:

$ virtualenv -p python3.6 ~/libertem-venv/
$ source ~/libertem-venv/bin/activate
(libertem) $ pip install libertem[torch]

Please see our documentation for details!