This is a library for building and querying a compressed form of set-membership filters, named k-XORSAT filters. These filters can be used similar to how one would use a Bloom filter but with one restriction --- items cannot be added after the filter is built. So, this is an 'offline' or 'static' filter, whereas Bloom filters are considered 'online' or 'dynamic'. The advantage is that k-XORSAT filters achieve very near the optimal memory usage. That is, they use much less memory than Bloom filters, making them desirable for either large data sets (save space) or data sets that are provided to a large user base (save bandwidth).
The advantage of k-XORSAT filters over other filters with small memory footprint is that the near-optimal compression of k-XORSAT filters is reliably achieved, and the query speed is comparable to Bloom filters for high false positive rates, and faster than Bloom filters for small false positive rates.
This package depends on pthreads and standard C math libraries.
This project also relies on a few git submodules. To get these modules, clone the repository by doing either
git clone --recursive email@example.com:NationalSecurityAgency/XORSATFilter.git
git clone firstname.lastname@example.org:NationalSecurityAgency/XORSATFilter.git cd XORSATFilter git submodule update --init --recursive
make in the project root directory. The library file
libxorsatfilter.a will (assuming successful compilation) be created
in this package's
k-XORSAT filters are built in two separate phases. The first step is to add elements to a builder object. The second phase is to create a querier object from a builder object. Once completed, the querier object may be queried ad infinitum.
A builder is first allocated using
XORSATFilterBuilder *xsfb = XORSATFilterBuilderAlloc(0, 0);
Here, the first argument
0 is the number of expected elements the
filter will encode. It is safe to leave this number as
0, but will
decrease calls to malloc if the number is given ahead of time. The
second argument is the number of bytes of metatdata to store with each
element. To use the data structure as a filter, enter 0. To use the
data structure as a dictionary, enter the number of bytes to store in
Elements are added to the builder, like so:
XORSATFilterBuilderAddElement(xsfb, pElement, nElementBytes, pMetaData);
pElement is a pointer to at least
nElementBytes number of
bytes. This element will be copied into the builder.
pMetaData is a
pointer to an array of bytes that will be stored into the data
structure and can be retrieved after construction. If the data
structure is used as a filter, this last argument may be
It is also possible to add the absence of an element to the
filter. This will cause the filter to return
False when queried
against the element.
XORSATFilterBuilderAddAbsence(xsfb, pElement, nElementBytes);
After all elements have been stored, the querier is ready to be created:
XORSATFilterQuerier *xsfq = XORSATFilterBuilderFinalize(xsfb, XORSATFilterFastParameters, nThreads);
The second argument is a structure consisting of four parameters: the number of literals per row, the number of solutions, the number of elements per block, and the desired efficiency of the filter. Three samples are provided by the package,
XORSATFilterEfficientParameterscreates small filters but has longer build time.
XORSATFilterPaperParameterscreates filters according to parameters used in a recent paper on XORSAT filters.
XORSATFilterFastParameterscreates filters quickly but the filters are larger.
More information can be found near the tops of
src/xorsat_blocks.c. Feel free to
define your own parameters to meet the needs of your application.
nThreads argument corresponds to the number of pthreads
used when building the querier. The returned querier (
xsfq) will be
NULL on error.
When finalizing, you will notice that the progress is printed to stderr. These print statements can be turned off by commenting out the following line in xorsat_filter.h and recompiling the package.
After creating the querier, it is suggested that the builder be free'd, like so:
The filter can be queried, like so:
uint8_t ret = XORSATFilterQuery(xsfq, pElement, nElementBytes);
pElement is a pointer to
nElementBytes number of bytes. If
this element might be in the filter (depending on the false positive
ret will return
ret will return
indicating that this element is definitely not in the filter.
Stored metadata can be retrived like so:
uint8_t *pMetaData_retrieved = XORSATFilterRetrieveMetadata(xsfq, pElement, nElementBytes);
pElement was not added to the data structure, the metadata
returned will appear random. Otherwise, the stored metadata will be
returned via a newly allocated pointer.
Queriers can be serialized (written to a file) in the following way:
uint8_t ret = XORSATFilterSerialize(fout, xsfq);
fout is of type
ret will be
0 on failure and
A querier can be deserialized (read from a file) in the following way:
xsfq = XORSATFilterDeserialize(fout);
fout is of type
xsfq will be
NULL on error.
When querying is done, the filter can be freed, like so:
To use, simply link against
lib/libxorsatfilter.a and include
This interface is demonstrated in a test provided with this library.
A sample interface is given in the
test directory. The test builds a
k-XORSAT dictionary for 1000000 random 10-byte elements each with
10-bytes of metadata, writes the dictionary to a file, reads the file
back, then queries the filter against the original elements (for a
consistency check) and prints statistics. To run the test type:
$ make test/test && test/test
A paper about SAT Filters is published in JSAT, available here: Satisfiability-based Set Membership Filters
A paper about XORSAT Filters is published in SAT'18, available here: XOR-Satisfiability Set Membership Filters
This work was prepared by U.S. Government employees and, therefore, is excluded from copyright by Section 105 of the Copyright Act of 1976.
This Work is provided "as is." Any express or implied warranties, including but not limited to, the implied warranties of merchantability and fitness for a particular purpose are disclaimed. In no event shall the United States Government be liable for any direct, indirect, incidental, special, exemplary or consequential damages (including, but not limited to, procurement of substitute goods or services, loss of use, data or profits, or business interruption) however caused and on any theory of liability, whether in contract, strict liability, or tort (including negligence or otherwise) arising in any way out of the use of this Guidance, even if advised of the possibility of such damage.
The User of this Work agrees to hold harmless and indemnify the United States Government, its agents and employees from every claim or liability (whether in tort or in contract), including attorneys' fees, court costs, and expenses, arising in direct consequence of Recipient's use of the item, including, but not limited to, claims or liabilities made for injury to or death of personnel of User or third parties, damage to or destruction of property of User or third parties, and infringement or other violations of intellectual property or technical data rights.
Nothing in this Work is intended to constitute an endorsement, explicit or implied, by the U.S. Government of any particular manufacturer's product or service.