Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make the acceptable compression overhead used by MultiOrdinals configurable and default to FASTEST #3623

Closed
bleskes opened this Issue Sep 5, 2013 · 0 comments

Comments

Projects
None yet
1 participant
@bleskes
Copy link
Member

commented Sep 5, 2013

We currently store ordinal in a compressed fashion using Lucene's PackedInt structures. Currently we use the COMPACT compression setting which tries to save on memory as much as possible at the expense of CPU and execution time. To reduce the CPU cost, we will default to FASTEST compression but make it configurable so people can change it if needed, via the mapping API:

curl -XPUT "http://localhost:9200/index/type/_mapping" -d'
{
   "type": {
      "properties": {
         "field": {
            "type": "string",
            "fielddata": {
               "acceptable_overhead_ratio": 0.2 
            }
         }
      }
   }
}'

@ghost ghost assigned bleskes Sep 5, 2013

@bleskes bleskes closed this in c6ac5ac Sep 5, 2013

bleskes added a commit that referenced this issue Sep 5, 2013

Make the acceptable compression overhead used by MultiOrdinals config…
…urable and default to PackedInts.FASTEST (causing it to byte align).

Closes #3623

Before this commit , this was the output of TermsFacetSearchBenchmark, on my MacBookAir:

```
------------------ SUMMARY -------------------------------
                     name      took    millis
                  terms_s      7.3s        36
              terms_map_s     28.8s       144
                  terms_l     15.9s        79
              terms_map_l     15.5s        77
                 terms_sm        1m       319
             terms_map_sm      4.9m      1491
                 terms_lm      2.7m       825
             terms_map_lm      2.7m       829
          terms_stats_s_l     37.6s       188
         terms_stats_s_lm      2.4m       722
         terms_stats_sm_l      6.5m      1958
------------------ SUMMARY -------------------------------
```

After the change to FASTEST, we have:

```
------------------ SUMMARY -------------------------------
                     name      took    millis
                  terms_s      6.9s        34
              terms_map_s     28.8s       144
                  terms_l     17.4s        87
              terms_map_l     17.6s        88
                 terms_sm       42s       210
             terms_map_sm      4.2m      1287
                 terms_lm      2.3m       714
             terms_map_lm      2.3m       716
          terms_stats_s_l     37.5s       187
         terms_stats_s_lm      1.6m       482
         terms_stats_sm_l      6.1m      1852
------------------ SUMMARY -------------------------------
```

mute pushed a commit to mute/elasticsearch that referenced this issue Jul 29, 2015

Make the acceptable compression overhead used by MultiOrdinals config…
…urable and default to PackedInts.FASTEST (causing it to byte align).

Closes elastic#3623

Before this commit , this was the output of TermsFacetSearchBenchmark, on my MacBookAir:

```
------------------ SUMMARY -------------------------------
                     name      took    millis
                  terms_s      7.3s        36
              terms_map_s     28.8s       144
                  terms_l     15.9s        79
              terms_map_l     15.5s        77
                 terms_sm        1m       319
             terms_map_sm      4.9m      1491
                 terms_lm      2.7m       825
             terms_map_lm      2.7m       829
          terms_stats_s_l     37.6s       188
         terms_stats_s_lm      2.4m       722
         terms_stats_sm_l      6.5m      1958
------------------ SUMMARY -------------------------------
```

After the change to FASTEST, we have:

```
------------------ SUMMARY -------------------------------
                     name      took    millis
                  terms_s      6.9s        34
              terms_map_s     28.8s       144
                  terms_l     17.4s        87
              terms_map_l     17.6s        88
                 terms_sm       42s       210
             terms_map_sm      4.2m      1287
                 terms_lm      2.3m       714
             terms_map_lm      2.3m       716
          terms_stats_s_l     37.5s       187
         terms_stats_s_lm      1.6m       482
         terms_stats_sm_l      6.1m      1852
------------------ SUMMARY -------------------------------
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.