Skip to content

posterior/loom

master
Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?
Code
This branch is 25 commits ahead of priorknowledge:master.

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
doc
 
 
 
 
 
 
 
 
 
 
src
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Loom

Build Status DOI

Loom is a streaming inference and query engine for the Cross-Categorization model mansinghka2009cross, shafto2011probabilistic.

Data Types

Loom learns models of sparse heterogeneous tabular data, with hundreds of features and millions of rows. Loom currently supports the following feature types and models:

  • boolean fields as Beta-Bernoulli
  • categorical fields with up to 256 values as Dirichlet-Discrete
  • unbounded categorical fields as Dirichlet-Process-Discrete
  • count fields as Gamma-Poisson
  • real fields as Normal-Inverse-Chi-Squared-Normal
  • sparse real fields as mixture of degenerate and dense real
  • text and keyword fields as booleans for word absence/presence
  • date fields as a combination of absolute, relative, and cyclic parts
  • optional fields as a boolean plus one of the above feature models

See input format docs for details.

Data Scale

Loom targets tabular datasets of sizes 100-1000 columns 10^3-10^9 rows. To handle large datasets, loom implements subsample annealing obermeyer2014scaling with an accelerating annealing schedule and adaptively turns off ineffective inference strategies. Loom's annealing schedule is tuned to learn 10^8 cell datasets in under an hour and 10^10 cell datasets in under a day (depending on feature type and sparsity).

   Full Inference:     Partial Inference:  Greedy Inference:
   structure
   hyperparameters     hyperparameters
   mixtures            mixtures            mixtures
 |-------------------> ------------------> ------------------>
 1   many-passes   ~10^4   accelerate   10^9   single-pass  10^4
row                 rows                rows               row/sec

Documentation

Authors

Loom is a streaming rewrite of the TARDIS engine developed by Eric Jonas https://twitter.com/stochastician at Prior Knowledge, Inc.

Loom relies heavily on the distributions library.

License

Copyright (c) 2014 Salesforce.com, Inc. All rights reserved. Copyright (c) 2015, Google, Inc.

Licensed under the Revised BSD License. See LICENSE.txt for details.

The PreQL query interface is covered by US patents pending:

  • Application No. 14/014,204
  • Application No. 14/014,221
  • Application No. 14/014,225
  • Application No. 14/014,236
  • Application No. 14/014,241
  • Application No. 14/014,250
  • Application No. 14/014,258

Dependencies

Packages

No packages published

Languages

  • C++ 54.8%
  • Python 44.3%
  • Other 0.9%