Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: document in-memory joins in optimization docs #742

Merged
merged 3 commits into from
Mar 21, 2019
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 10 additions & 10 deletions Gopkg.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Gopkg.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[[constraint]]
name = "gopkg.in/src-d/go-mysql-server.v0"
revision = "b32d2fdea095e2743d13f3ab4da5ae83aef55bc7"
revision = "0093a7562ad1cf31f179396dfa5be32893059dbb"

[[constraint]]
name = "github.com/jessevdk/go-flags"
Expand Down
2 changes: 1 addition & 1 deletion docs/using-gitbase/functions.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,4 +95,4 @@ Also, if you want to retrieve values from a non common property, you can pass it

## Standard functions

You can check standard functions in [`go-mysql-server` documentation](https://github.com/src-d/go-mysql-server/tree/b32d2fdea095e2743d13f3ab4da5ae83aef55bc7#custom-functions).
You can check standard functions in [`go-mysql-server` documentation](https://github.com/src-d/go-mysql-server/tree/0093a7562ad1cf31f179396dfa5be32893059dbb#custom-functions).
2 changes: 1 addition & 1 deletion docs/using-gitbase/indexes.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,4 +26,4 @@ and for the second query also two indexes will be used and the result will be a

You can find some more examples in the [examples](./examples.md#create-an-index-for-columns-on-a-table) section.

See [go-mysql-server](https://github.com/src-d/go-mysql-server/tree/b32d2fdea095e2743d13f3ab4da5ae83aef55bc7#indexes) documentation for more details
See [go-mysql-server](https://github.com/src-d/go-mysql-server/tree/0093a7562ad1cf31f179396dfa5be32893059dbb#indexes) documentation for more details
19 changes: 19 additions & 0 deletions docs/using-gitbase/optimize-queries.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ Even though in each release performance improvements are included to make gitbas
There are two ways to optimize a gitbase query:
- Create an index for some parts.
- Making sure the joined tables are squashed.
- Making sure not squashed joins are performed in memory.

## Assessing performance bottlenecks

Expand Down Expand Up @@ -57,6 +58,24 @@ Some performance issues might not be obvious, but there are a few that really st

- Joins not squashed. If you performed some joins between tables and instead of a `SquashedTable` node you see `Join` and `Table` nodes, it means the joins were not successfully squashed. There is a more detailed explanation about this in next sections of this document.
- Indexes not used. If you can't see the indexes in your table nodes, it means somehow those indexes are not being used by the table. There is a more detailed explanation about this in next sections of this document.
- Joins not squashed that are not being executed in memory. There is a more detailed explanation about this in the next sections of this document.

## In-memory joins

There are two modes in which gitbase can execute an inner join:

- Multipass: it fully iterates the right side of the join one time for each row in the left side. This is really expensive, but avoids having to load one side fully in memory.
- In-memory: loads the whole right side in memory and iterates the left side. Both sides are iterated exactly once, thus it makes the query much faster, but it has the disadvantage of potentially requiring a lot of memory.

The default mode is multipass, unless the right side fits in memory (there's a more elaborate explanation about this below).

In-memory joins can be enabled at the user request, either with the `EXPERIMENTAL_IN_MEMORY_JOIN=on` environment variable or executing `SET inmemory_joins = 1`. The last method only enables it for the current connection.

Even if they are not globally enabled for all queries, there is an optimization that checks if the join could be performed in memory and if it can't, switches to multipass mode.
As long as the whole gitbase server memory usage is under the 20% of all available physical (not counting other memory used by other processes) memory in the machine, the join will be performed in memory. When this limit is passed, the multipass mode will be used instead.
20% is just a default value that can be changed using the `MAX_MEMORY_INNER_JOIN` environment variable to the maximum amount of bytes the gitbase server can be using before switching to multipass mode. It can also be changed per session using `SET max_memory_joins=<MAX BYTES>`.

So, as a good rule of thumb, the right side of an inner join should always be the smaller one, because that way, it has bigger chances of being executed in memory and it will be faster.

## Indexes

Expand Down
2 changes: 1 addition & 1 deletion docs/using-gitbase/supported-clients.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
## Supported clients

To see the supported MySQL clients and examples about how to use them, take a look [here](https://github.com/src-d/go-mysql-server/blob/b32d2fdea095e2743d13f3ab4da5ae83aef55bc7/SUPPORTED_CLIENTS.md).
To see the supported MySQL clients and examples about how to use them, take a look [here](https://github.com/src-d/go-mysql-server/blob/0093a7562ad1cf31f179396dfa5be32893059dbb/SUPPORTED_CLIENTS.md).
2 changes: 1 addition & 1 deletion docs/using-gitbase/supported-syntax.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
## Supported syntax

To see the SQL subset currently supported take a look at [this list](https://github.com/src-d/go-mysql-server/blob/b32d2fdea095e2743d13f3ab4da5ae83aef55bc7/SUPPORTED.md) from [src-d/go-mysql-server](https://github.com/src-d/go-mysql-server).
To see the SQL subset currently supported take a look at [this list](https://github.com/src-d/go-mysql-server/blob/0093a7562ad1cf31f179396dfa5be32893059dbb/SUPPORTED.md) from [src-d/go-mysql-server](https://github.com/src-d/go-mysql-server).
1 change: 0 additions & 1 deletion vendor/github.com/moovweb/rubex/AUTHOR

This file was deleted.

19 changes: 0 additions & 19 deletions vendor/github.com/moovweb/rubex/LICENSE

This file was deleted.

38 changes: 0 additions & 38 deletions vendor/github.com/moovweb/rubex/README

This file was deleted.

1 change: 0 additions & 1 deletion vendor/github.com/moovweb/rubex/VERSION

This file was deleted.

Loading