Skip to content

Update benchmarks, run them, check in results as markdown files #18

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Mar 24, 2021
Merged

Update benchmarks, run them, check in results as markdown files #18

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
af2369c
update benchmarks, run all but the all one and check in results
kevinlang Mar 24, 2021
1ac9ab9
format
kevinlang Mar 24, 2021
21ac763
clean up lockfile
kevinlang Mar 24, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .formatter.exs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
[
inputs: [
"{mix,.formatter}.exs",
"{lib,test}/**/*.{ex,exs}"
"{lib,test,bench}/**/*.{ex,exs}"
],
line_length: 88
]
6 changes: 6 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,3 +39,9 @@ config :my_app,
config :my_app, MyApp.Repo,
database: "path/to/my/database.db",
```

## Benchmarks

We have some benchmarks comparing it against the `MySQL` and `Postgres` adapters.

You can read more about those at [bench/README.md](bench/README.md).
19 changes: 16 additions & 3 deletions bench/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,17 @@
# Ecto Benchmarks

## Results

| Benchmark | Description |
| --------- | ------------ |
| [load](results/load.md) | tracks performance of loading "raw" data into ecto structures |
| [to_sql](results/to_sql.md) | tracks performance of parsing `Ecto.Query` structures into "raw" SQL query strings |
| [insert](results/insert.md) | tracks performance of inserting changesets and structs in the database with `Repo.insert!/1` function |

For reference, these results were run with a Sabrent Rocket Q 1TB NVMe SSD.

## Running the benchmarks

Ecto has a benchmark suite to track performance of sensitive operations. Benchmarks
are run using the [Benchee](https://github.com/PragTob/benchee) library and
need PostgreSQL and MySQL up and running.
Expand Down Expand Up @@ -28,9 +40,10 @@ more likely to integration tests.
You can also run a benchmark individually by giving the path to the benchmark
script instead of `bench/bench_helper.exs`.

# Docker
I had Postgres already installed and running locally, but needed to get MySQL up and running. The easiest way to do this is with this command:
### Docker
The easiest way to setup mysql and postgresql for the benchmarks is via Docker. Run the following commands to get an instance of each running.

```
docker run -p 3306:3306 --name mysql_server -e MYSQL_ALLOW_EMPTY_PASSWORD=yes mysql:5.7
docker run -p 3306:3306 -e MYSQL_ALLOW_EMPTY_PASSWORD=yes mysql:5.7
docker run -p 5432:5432 -e POSTGRES_PASSWORD=postgres postgres:13.2
```
204 changes: 204 additions & 0 deletions bench/results/insert.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,204 @@

# Benchmark

Benchmark run from 2021-03-24 02:05:58.706995Z UTC

## System

Benchmark suite executing on the following system:

<table style="width: 1%">
<tr>
<th style="width: 1%; white-space: nowrap">Operating System</th>
<td>Linux</td>
</tr><tr>
<th style="white-space: nowrap">CPU Information</th>
<td style="white-space: nowrap">AMD Ryzen 7 PRO 4750U with Radeon Graphics</td>
</tr><tr>
<th style="white-space: nowrap">Number of Available Cores</th>
<td style="white-space: nowrap">16</td>
</tr><tr>
<th style="white-space: nowrap">Available Memory</th>
<td style="white-space: nowrap">14.92 GB</td>
</tr><tr>
<th style="white-space: nowrap">Elixir Version</th>
<td style="white-space: nowrap">1.11.3</td>
</tr><tr>
<th style="white-space: nowrap">Erlang Version</th>
<td style="white-space: nowrap">23.2.6</td>
</tr>
</table>

## Configuration

Benchmark suite executing with the following configuration:

<table style="width: 1%">
<tr>
<th style="width: 1%">:time</th>
<td style="white-space: nowrap">5 s</td>
</tr><tr>
<th>:parallel</th>
<td style="white-space: nowrap">1</td>
</tr><tr>
<th>:warmup</th>
<td style="white-space: nowrap">2 s</td>
</tr>
</table>

## Statistics




__Input: Changeset__

Run Time

<table style="width: 1%">
<tr>
<th>Name</th>
<th style="text-align: right">IPS</th>
<th style="text-align: right">Average</th>
<th style="text-align: right">Devitation</th>
<th style="text-align: right">Median</th>
<th style="text-align: right">99th&nbsp;%</th>
</tr>

<tr>
<td style="white-space: nowrap">SQLite3 Insert</td>
<td style="white-space: nowrap; text-align: right">7218.07</td>
<td style="white-space: nowrap; text-align: right">0.139 ms</td>
<td style="white-space: nowrap; text-align: right">±43.60%</td>
<td style="white-space: nowrap; text-align: right">0.123 ms</td>
<td style="white-space: nowrap; text-align: right">0.37 ms</td>
</tr>

<tr>
<td style="white-space: nowrap">Pg Insert</td>
<td style="white-space: nowrap; text-align: right">421.57</td>
<td style="white-space: nowrap; text-align: right">2.37 ms</td>
<td style="white-space: nowrap; text-align: right">±12.13%</td>
<td style="white-space: nowrap; text-align: right">2.37 ms</td>
<td style="white-space: nowrap; text-align: right">2.90 ms</td>
</tr>

<tr>
<td style="white-space: nowrap">MyXQL Insert</td>
<td style="white-space: nowrap; text-align: right">284.25</td>
<td style="white-space: nowrap; text-align: right">3.52 ms</td>
<td style="white-space: nowrap; text-align: right">±13.34%</td>
<td style="white-space: nowrap; text-align: right">3.53 ms</td>
<td style="white-space: nowrap; text-align: right">5.05 ms</td>
</tr>

</table>


Comparison

<table style="width: 1%">
<tr>
<th>Name</th>
<th style="text-align: right">IPS</th>
<th style="text-align: right">Slower</th>
<tr>
<td style="white-space: nowrap">SQLite3 Insert</td>
<td style="white-space: nowrap;text-align: right">7218.07</td>
<td>&nbsp;</td>
</tr>

<tr>
<td style="white-space: nowrap">Pg Insert</td>
<td style="white-space: nowrap; text-align: right">421.57</td>
<td style="white-space: nowrap; text-align: right">17.12x</td>
</tr>

<tr>
<td style="white-space: nowrap">MyXQL Insert</td>
<td style="white-space: nowrap; text-align: right">284.25</td>
<td style="white-space: nowrap; text-align: right">25.39x</td>
</tr>

</table>



<hr/>


__Input: Struct__

Run Time

<table style="width: 1%">
<tr>
<th>Name</th>
<th style="text-align: right">IPS</th>
<th style="text-align: right">Average</th>
<th style="text-align: right">Devitation</th>
<th style="text-align: right">Median</th>
<th style="text-align: right">99th&nbsp;%</th>
</tr>

<tr>
<td style="white-space: nowrap">SQLite3 Insert</td>
<td style="white-space: nowrap; text-align: right">7765.76</td>
<td style="white-space: nowrap; text-align: right">0.129 ms</td>
<td style="white-space: nowrap; text-align: right">±32.88%</td>
<td style="white-space: nowrap; text-align: right">0.122 ms</td>
<td style="white-space: nowrap; text-align: right">0.28 ms</td>
</tr>

<tr>
<td style="white-space: nowrap">Pg Insert</td>
<td style="white-space: nowrap; text-align: right">422.86</td>
<td style="white-space: nowrap; text-align: right">2.36 ms</td>
<td style="white-space: nowrap; text-align: right">±10.49%</td>
<td style="white-space: nowrap; text-align: right">2.36 ms</td>
<td style="white-space: nowrap; text-align: right">3.02 ms</td>
</tr>

<tr>
<td style="white-space: nowrap">MyXQL Insert</td>
<td style="white-space: nowrap; text-align: right">274.00</td>
<td style="white-space: nowrap; text-align: right">3.65 ms</td>
<td style="white-space: nowrap; text-align: right">±38.43%</td>
<td style="white-space: nowrap; text-align: right">3.59 ms</td>
<td style="white-space: nowrap; text-align: right">4.75 ms</td>
</tr>

</table>


Comparison

<table style="width: 1%">
<tr>
<th>Name</th>
<th style="text-align: right">IPS</th>
<th style="text-align: right">Slower</th>
<tr>
<td style="white-space: nowrap">SQLite3 Insert</td>
<td style="white-space: nowrap;text-align: right">7765.76</td>
<td>&nbsp;</td>
</tr>

<tr>
<td style="white-space: nowrap">Pg Insert</td>
<td style="white-space: nowrap; text-align: right">422.86</td>
<td style="white-space: nowrap; text-align: right">18.37x</td>
</tr>

<tr>
<td style="white-space: nowrap">MyXQL Insert</td>
<td style="white-space: nowrap; text-align: right">274.00</td>
<td style="white-space: nowrap; text-align: right">28.34x</td>
</tr>

</table>



<hr/>

Loading