Recommendation on ID indexes

[GoogleCodeExporter]

Hi,

We have structure ( Trade ) which has ID fields unique always.
We noticed if use Hash or Navigable index on ID field to push it into CqEngine 
collection the size of heap taken is doubled up, which is baffling in a way. 
SO basically adding a index where values are going to Unique is not working out 
well for us. DO you have any recommendation on which index to be used and how 
we can reduce the memory footprint here ?
The more indexes the heap size is getting lot worst, specially if the values 
are unique and spread is high.

Any recommendations are highly appreciated. 

Thanks,
Sandeep

Original issue reported on code.google.com by sandeepd...@gmail.com on 15 Oct 2013 at 6:58

[GoogleCodeExporter]

Hi Sandeep, this belongs more in the forum than as an issue.

Have you tried IndexQuantization? This is a good way to reduce memory usage.

Also are your ID values sparse? If so you could write your own quantizer to 
better handle sparse values, like this one:

/**
 * Converts sparse double values, to a fixed number of buckets.
 * <p/>
 * This quantizer should only be used with equality-based indexes, such as {@code HashIndex}.
 */
public class SparseDoubleQuantizer implements Quantizer<Double> {
    private final int numBuckets;
    public SparseDoubleQuantizer(int numBuckets) {
        this.numBuckets = numBuckets;
    }
    @Override
    public Double getQuantizedValue(Double attributeValue) {
        return (double) attributeValue.longValue() % numBuckets;
    }
}

The quantizer above will restrict the size of an equality-based index (e.g. 
HashIndex) to a fixed number of buckets, independent of the number of unique 
values of your IDs. But it cannot be used with a NavigableIndex.

This is not the best place to answer questions. If you like post your question 
in the forum http://groups.google.com/group/cqengine-discuss and we can 
continue discussion there. thanks!

Original comment by ni...@npgall.com on 15 Oct 2013 at 12:10

• Changed state: Invalid
• Added labels: Type-Other
• Removed labels: Type-Defect