This repository has been archived by the owner on Jun 28, 2024. It is now read-only.
core during computing join #31
Comments
|
The join is already multi-threaded. However, this depends on the query, and the query plan. If the joins are organised in a sequence, then there is no possibility for the query optimizer to execute them in parallel. |
|
To be more precise, there could be some additional optimisation that the query optimizer could do to try to execute more joins in parallel, but this is not something that will be available immediately. Also we are working on a new distributed computing platform for joins that will leverage better the available computing nodes. |
|
Here the logs when i refresh my dashboard Veicoli (means Cars) with many relations. |
|
You can set the log level to DEBUG in kibi.yml Stéphane Campinas
|
|
You have also in the query response form elasticsearch some statistics about each join, where you can see the execution time, size, etc, of each join. This might indicate you which one is taking time. If you have enough memory on your node, you might want to tune the node-level cache [1] for the siren join plugin. |
|
@scampi in kibi.yml i can only set logging.verbose: true or i have to set something else? @rendel this is the answer to the stats: I have also enabled caching (size set to 512 MB) as you suggest me but i can't see any improvment... |
|
@scampi I think the debug logging does not work in kibi |
|
@bertol83 the cache is empty for some reason ( |
|
Here my configuration in elasticsearch.yml Here the stats after some requests: {
"cluster_name": "aci-roma",
"nodes": {
"Free Spirit": {
"timestamp": 1464074644278,
"stats": {
"size": 0,
"requestCount": 24,
"hitCount": 0,
"hitRate": 0,
"missCount": 24,
"missRate": 1,
"loadCount": 0,
"loadSuccessCount": 0,
"loadExceptionCount": 0,
"loadExceptionRate": 0,
"totalLoadTime": 0,
"evictionCount": 24
}
}
}
}Here kibi logs when i press F5: Here the elasticsearch logs: |
|
The cache is ever empty, and the time to compute joins is always 70 seconds. Maybe this is the limit because joins are very big. Index Cars (10M documents) is in join with: Maybe i could try to use a second node for elasticsearch, what do you think? |
|
Adding a second node would probably help in distributing the load. Is the server using ssd or spinning disks ? |
|
After investigation, we discovered that the performance issue was coming from the cache size setting. The siren join is setting an incorrect default cache size of 256kb (instead of 256mb). An issue has been opened to resolve this in the next siren-join release. solved the issue. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
To improve performance and it would be possible make computing joins multi core? If you look my screenshot there is just one core running, but joins in this case are 7.
The text was updated successfully, but these errors were encountered: