Skip to content
This repository was archived by the owner on Jul 3, 2023. It is now read-only.

Conversation

@lalithsuresh
Copy link
Contributor

  • Conditionally build pg-embed use outside tests using a "dev" feature flag, enabled by default
  • Separate PgEmbed out from ProjectDB
  • Update Docker build to use non-default builds

* Conditionally build pg-embed use outside tests using a "dev" feature flag, enabled by default
* Separate PgEmbed out from ProjectDB
* Update Docker build to use non-default builds

Signed-off-by: Lalith Suresh <lsuresh@vmware.com>
Signed-off-by: Lalith Suresh <lsuresh@vmware.com>
@lalithsuresh lalithsuresh requested a review from gz April 28, 2023 18:16
@codecov
Copy link

codecov bot commented Apr 28, 2023

Codecov Report

Merging #384 (bba3d22) into main (dbe4663) will increase coverage by 0.13%.
The diff coverage is 70.09%.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #384      +/-   ##
==========================================
+ Coverage   72.99%   73.12%   +0.13%     
==========================================
  Files         237      237              
  Lines       49568    50156     +588     
==========================================
+ Hits        36180    36675     +495     
- Misses      13388    13481      +93     
Impacted Files Coverage Δ
crates/dataflow-jit/src/codegen/intrinsics.rs 61.60% <0.00%> (-8.93%) ⬇️
crates/dataflow-jit/src/dataflow/mod.rs 49.88% <0.00%> (-10.08%) ⬇️
crates/dataflow-jit/src/ir/function/passes.rs 52.57% <ø> (ø)
crates/dataflow-jit/src/ir/ids.rs 61.29% <0.00%> (-2.05%) ⬇️
crates/dataflow-jit/src/ir/nodes/aggregate.rs 10.48% <ø> (-0.40%) ⬇️
crates/dataflow-jit/src/ir/nodes/constant.rs 47.74% <ø> (+1.25%) ⬆️
crates/dataflow-jit/src/ir/nodes/differentiate.rs 17.18% <ø> (+1.47%) ⬆️
crates/dataflow-jit/src/ir/nodes/filter_map.rs 41.62% <ø> (+0.32%) ⬆️
crates/dataflow-jit/src/ir/nodes/flat_map.rs 81.13% <ø> (-1.02%) ⬇️
crates/dataflow-jit/src/ir/nodes/index.rs 57.57% <ø> (-1.85%) ⬇️
... and 41 more

... and 3 files with indirect coverage changes

Signed-off-by: Lalith Suresh <lsuresh@vmware.com>

fn default_db_connection_string() -> String {
"postgres-embed".to_string()
"".to_string()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be postgres:// now or something similar?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tokio-postgres doesn't fill things in automatically, so "postgres://" does not work.

use pg_embed::pg_fetch::{PgFetchSettings, PG_V15};
use pg_embed::postgres::{PgEmbed, PgSettings};
use std::path::PathBuf;
#[cfg(feature = "dev")]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you instead make the

mod pg_setup;

in mod.rs

as

#[cfg(feature = "dev")]
mod pg_setup;

you wont have to make an extra module (embedded) for it.

ConnectorType, PipelineId, ProjectDB, ProjectDescr, ProjectId, ProjectStatus, Version,
};
use crate::db::DBError;
use crate::db::{pg_setup, DBError};
Copy link
Contributor

@gz gz Apr 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so mod test in mod.rs maybe also needs to be conditionally included with cfg(feature = dev)?

or cargo test --no-default-features might fail?

Copy link
Contributor Author

@lalithsuresh lalithsuresh Apr 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've updated the CI workflows to have feature=dev wherever --no-default-features is called. It's better to fail the run than to silently not run the tests.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe it will result in a compilation error if you do cargo test --no-default-features which is a bit awkward. You could update to include the crate::db::pg_setup cfg(feature = "dev") and in addition also for cfg(test)?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

e.g., cfg(any(test, feature = "dev") might do the trick?

Signed-off-by: Lalith Suresh <lsuresh@vmware.com>
@github-actions
Copy link

github-actions bot commented Apr 29, 2023

Benchmark results

Nexmark

  • 1 out of 21 queries have regressed ❗
  • Compared results from dbe4663 (main) with bba3d22 (PR)
    No benchmark results found for current main revision, compared against dbe4663
name main~15 [kOp/s] PR [kOp/s] Tput change [%] Assessment Peak RSS diff
q0 5262.63 5396.07 3 ✔️ 2.9 MB
q1 5429.6 5386.13 -1 ✔️ 4.1 MB
q2 5361.6 5381.73 0 ✔️ -3.6 MB
q3 5321.94 5290.19 -1 ✔️ 28.2 MB
q4 4802.55 4832.96 1 ✔️ -329.7 MB
q5 5378.33 5284.93 -2 ✔️ -329.7 MB
q6 4880.13 4799.99 -2 ✔️ -184.4 MB
q7 2559.31 2855.92 12 🌲 -1.2 GB
q8 5161.2 5203.35 1 ✔️ -1.2 GB
q9 875.448 884.805 1 ✔️ -5.9 MB
q12 4910.28 4949.29 1 ✔️ -5.9 MB
q13 3645.17 2789.64 -23 🔻 -5.9 MB
q14 5481.43 5415.94 -1 ✔️ -5.9 MB
q15 4895.11 5417.9 11 🌲 -5.9 MB
q16 1045.16 1046.76 0 ✔️ -5.9 MB
q17 3134.56 3161.79 1 ✔️ 53.4 MB
q18 1440.13 1436.34 0 ✔️ 1.2 GB
q19 1435.15 1420.83 -1 ✔️ 1.2 GB
q20 1500.83 1558.97 4 ✔️ 1.2 GB
q21 5117.36 5254.3 3 ✔️ 1.2 GB
q22 5356.59 5420.18 1 ✔️ 1.2 GB

Galen

  • Compared results from dbe4663 (main) with bba3d22 (PR)
    No benchmark results found for current main revision, compared against dbe4663
name main~15 [s] PR [s] Runtime change [%] Assessment
galen 28.6952 28.755 0 ✔️

LDBC

  • 1 out of 4 queries have regressed ❗
  • Compared results from dbe4663 (main) with bba3d22 (PR)
    No benchmark results found for current main revision, compared against dbe4663
algorithm dataset threads main~15 [kEVPS] PR [kEVPS] Tput change [%] Assessment Peak RSS diff
bfs graph500-22 1 1835.7 1797.89 -2 ✔️ -213.0 kB
bfs datagen-8_4-fb 6 8173.61 7667.11 -6 🔻 -22.4 MB
pagerank graph500-22 1 681.655 697.612 2 ✔️ 69.6 kB
pagerank datagen-8_4-fb 6 1994.79 2055.81 3 ✔️ -79.2 MB

Nexmark (with Persistence)

  • Compared results from dbe4663 (main) with bba3d22 (PR)
    No benchmark results found for current main revision, compared against dbe4663
name main~15 [kOp/s] PR [kOp/s] Tput change [%] PR DRAM [kOp/s] DRAM diff [%] Assessment
q0 2399.23 2482.98 3 2401.16 3 ✔️
q1 1713.93 1686.1 -2 1683.39 0 ✔️
q2 2407.67 2445.9 2 2347.67 4 ✔️
q3 2036.93 2074.79 2 2248.06 -8 ✔️
q4 360.256 362.443 1 1407.34 -74 ✔️
q5 2015.3 2064.37 2 2250.84 -8 ✔️
q6 332.421 337.455 2 1327.47 -75 ✔️
q7 639.584 671.523 5 1309.32 -49 ✔️
q8 2171.19 2263.23 4 2230.05 1 ✔️
q9 79.0764 79.6782 1 384.983 -79 ✔️
q12 838.481 873.33 4 1785.09 -51 ✔️
q13 443.491 453.723 2 1002.01 -55 ✔️
q14 1697.52 1696.67 0 1679.52 1 ✔️
q15 198.112 197.639 0 1195.3 -83 ✔️
q16 25.5771 26.1254 2 285.129 -91 ✔️
q17 81.4521 81.2814 0 791.462 -90 ✔️
q18 124.763 125.052 0 787.564 -84 ✔️
q19 182.489 181.27 -1 649.988 -72 ✔️
q20 505.761 508.185 0 936.528 -46 ✔️
q21 1489.8 1512.76 2 1512.2 0 ✔️
q22 2102.89 2028.14 -4 2057.76 -1 ✔️

…ther.

Signed-off-by: Lalith Suresh <lsuresh@vmware.com>
@lalithsuresh lalithsuresh merged commit e6cdbb5 into main Apr 30, 2023
@gz gz mentioned this pull request May 1, 2023
5 tasks
@lalithsuresh lalithsuresh deleted the k8s-postgres branch May 4, 2023 14:39
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants