Skip to content

Commit

Permalink
Rename all database identifiers to dashed variants
Browse files Browse the repository at this point in the history 
Having CamelCase names for tables and columns doesn't really work out
well, because we'd need to not only quote those names during creation
but also all queries would need to use double quotes around these
identifiers.

Now every database identifier is consistent and lower-case only, which
should make it easier to use within raw statements (and inside the SQL
shell) and also less confusing in terms of schema definition versus raw
statements or other code references.

Of course, this isn't a very trivial change, so I wrote a small script
which makes it easy to rebase this very commit should there be changes
to master in the meantime.

The Nix expression generating the script can be found here (yes, it's
gory and a variant of "write-once-never-touch-again"):

https://gist.github.com/aszlig/59ea813c28731d339b9764c46f363200

Note that I've left "userName" as-is (just making it lower-case in the
schema) because there were a lot of references in the code that overlap
with the column name.

Another table I didn't touch except from making it lower-case (which it
is anyway on PostgreSQL) is "schemaversion", because that table is
needed for upgrading existing instances.

Moreover, the reason why there now is no relation from jobset_inputs to
jobsets (and back) anymore in the DBIx schema is that because it is now
created using PostgreSQL instead of SQLite, that foreign key however was
defined for SQLite only:

> #ifdef SQLITE
>    ,
>    foreign key   (project, name, nixExprInput)
>       references JobsetInputs(project, jobset,
>       name)
> #endif

I've ran tests.install, tests.api and the tests in build.x86_64-linux
and they all have passed.

In addition to running those tests, I've dumped my running Hydra
instance with database schema version 45 and tested the change.

The upgrade worked successfully and the web interface is still working
after a few tests.

Signed-off-by: aszlig <aszlig@redmoonstudios.org>
  • Loading branch information
@aszlig
aszlig committed Jul 7, 2016
1 parent 51535d0 commit 28d10b4
Show file tree
Hide file tree
Showing 100 changed files with 1,921 additions and 1,248 deletions.
16 changes: 8 additions & 8 deletions doc/dev-notes.txt
View file
Expand Up @@ -80,12 +80,12 @@

* Restart all aborted builds in a given evaluation (e.g. 820909):

> update builds set finished = 0 where id in (select id from builds where finished = 1 and buildstatus = 3 and exists (select 1 from jobsetevalmembers where eval = 820909 and build = id));
> update builds set finished = 0 where id in (select id from builds where finished = 1 and build_status = 3 and exists (select 1 from jobset_eval_members where eval = 820909 and build = id));


* Restart all builds in a given evaluation that had a build step time out:

> update builds set finished = 0 where id in (select id from builds where finished = 1 and buildstatus != 0 and exists (select 1 from jobsetevalmembers where eval = 926992 and build = id) and exists (select 1 from buildsteps where build = id and status = 7));
> update builds set finished = 0 where id in (select id from builds where finished = 1 and build_status != 0 and exists (select 1 from jobset_eval_members where eval = 926992 and build = id) and exists (select 1 from build_steps where build = id and status = 7));


* select * from (select project, jobset, job, system, max(timestamp) timestamp from builds where finished = 1 group by project, jobset, job, system) x join builds y on x.timestamp = y.timestamp and x.project = y.project and x.jobset = y.jobset and x.job = y.job and x.system = y.system;
Expand All @@ -98,15 +98,15 @@
delete from builds where finished = 0 and not exists (select 1 from buildschedulinginfo s where s.id = builds.id and busy = 1);


* select x.project, x.jobset, x.job, x.system, x.id, x.timestamp, r.buildstatus, b.id, b.timestamp
* select x.project, x.jobset, x.job, x.system, x.id, x.timestamp, r.build_status, b.id, b.timestamp
from (select project, jobset, job, system, max(id) as id from Builds where finished = 1 group by project, jobset, job, system) as a_
natural join Builds x
natural join BuildResultInfo r
left join Builds b on b.id =
(select max(id) from builds c
natural join buildresultinfo r2
where x.project = c.project and x.jobset = c.jobset and x.job = c.job and x.system = c.system
and x.id > c.id and r.buildstatus != r2.buildstatus);
and x.id > c.id and r.build_status != r2.build_status);

* Using PostgreSQL (version 9.2 or newer is required):

Expand All @@ -115,7 +115,7 @@

* Find the builds with the highest number of build steps:

select id, (select count(*) from buildsteps where build = x.id) as n from builds x order by n desc;
select id, (select count(*) from build_steps where build = x.id) as n from builds x order by n desc;


* Evaluating the NixOS Hydra jobs:
Expand All @@ -126,14 +126,14 @@
* Show all the failing jobs/systems in the nixpkgs:stdenv jobset that
succeed in the nixpkgs:trunk jobset:

select job, system from builds b natural join buildresultinfo where project = 'nixpkgs' and jobset = 'stdenv' and iscurrent = 1 and finished = 1 and buildstatus != 0 and exists (select 1 from builds natural join buildresultinfo where project = 'nixpkgs' and jobset = 'trunk' and job = b.job and system = b.system and iscurrent = 1 and finished = 1 and buildstatus = 0) order by job, system;
select job, system from builds b natural join buildresultinfo where project = 'nixpkgs' and jobset = 'stdenv' and is_current = 1 and finished = 1 and build_status != 0 and exists (select 1 from builds natural join buildresultinfo where project = 'nixpkgs' and jobset = 'trunk' and job = b.job and system = b.system and is_current = 1 and finished = 1 and build_status = 0) order by job, system;


* Get all Nixpkgs jobs that have never built succesfully:

select project, jobset, job from builds b1
where project = 'nixpkgs' and jobset = 'trunk' and iscurrent = 1
where project = 'nixpkgs' and jobset = 'trunk' and is_current = 1
group by project, jobset, job
having not exists
(select 1 from builds b2 where b1.project = b2.project and b1.jobset = b2.jobset and b1.job = b2.job and finished = 1 and buildstatus = 0)
(select 1 from builds b2 where b1.project = b2.project and b1.jobset = b2.jobset and b1.job = b2.job and finished = 1 and build_status = 0)
order by project, jobset, job;
4 changes: 2 additions & 2 deletions src/hydra-queue-runner/builder.cc
View file
Expand Up @@ -352,7 +352,7 @@ State::StepResult State::doBuildStep(nix::ref<Store> destStore, Step::ptr step,
if (build2->finishedInDB) continue;
printMsg(lvlError, format("marking build %1% as failed") % build2->id);
txn.parameterized
("update Builds set finished = 1, buildStatus = $2, startTime = $3, stopTime = $4, isCachedBuild = $5 where id = $1 and finished = 0")
("update builds set finished = 1, build_status = $2, start_time = $3, stop_time = $4, is_cached_build = $5 where id = $1 and finished = 0")
(build2->id)
((int) (build2->drvPath != step->drvPath && result.buildStatus() == bsFailed ? bsDepFailed : result.buildStatus()))
(result.startTime)
Expand All @@ -365,7 +365,7 @@ State::StepResult State::doBuildStep(nix::ref<Store> destStore, Step::ptr step,
won't be built again. */
if (result.stepStatus != bsCachedFailure && result.canCache)
for (auto & path : step->drv.outputPaths())
txn.parameterized("insert into FailedPaths values ($1)")(path).exec();
txn.parameterized("insert into failed_paths values ($1)")(path).exec();

txn.commit();
}
Expand Down
42 changes: 21 additions & 21 deletions src/hydra-queue-runner/hydra-queue-runner.cc
View file
Expand Up @@ -206,7 +206,7 @@ void State::clearBusy(Connection & conn, time_t stopTime)
{
pqxx::work txn(conn);
txn.parameterized
("update BuildSteps set busy = 0, status = $1, stopTime = $2 where busy = 1")
("update build_steps set busy = 0, status = $1, stop_time = $2 where busy = 1")
((int) bsAborted)
(stopTime, stopTime != 0).exec();
txn.commit();
Expand All @@ -217,9 +217,9 @@ unsigned int State::allocBuildStep(pqxx::work & txn, Build::ptr build)
{
/* Acquire an exclusive lock on BuildSteps to ensure that we don't
race with other threads creating a step of the same build. */
txn.exec("lock table BuildSteps in exclusive mode");
txn.exec("lock table build_steps in exclusive mode");

auto res = txn.parameterized("select max(stepnr) from BuildSteps where build = $1")(build->id).exec();
auto res = txn.parameterized("select max(stepnr) from build_steps where build = $1")(build->id).exec();
return res[0][0].is_null() ? 1 : res[0][0].as<int>() + 1;
}

Expand All @@ -230,7 +230,7 @@ unsigned int State::createBuildStep(pqxx::work & txn, time_t startTime, Build::p
unsigned int stepNr = allocBuildStep(txn, build);

txn.parameterized
("insert into BuildSteps (build, stepnr, type, drvPath, busy, startTime, system, status, propagatedFrom, errorMsg, stopTime, machine) values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12)")
("insert into build_steps (build, stepnr, type, drv_path, busy, start_time, system, status, propagated_from, error_msg, stop_time, machine) values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12)")
(build->id)
(stepNr)
(0) // == build
Expand All @@ -246,7 +246,7 @@ unsigned int State::createBuildStep(pqxx::work & txn, time_t startTime, Build::p

for (auto & output : step->drv.outputs)
txn.parameterized
("insert into BuildStepOutputs (build, stepnr, name, path) values ($1, $2, $3, $4)")
("insert into build_step_outputs (build, stepnr, name, path) values ($1, $2, $3, $4)")
(build->id)(stepNr)(output.first)(output.second.path).exec();

return stepNr;
Expand All @@ -260,7 +260,7 @@ void State::finishBuildStep(pqxx::work & txn, time_t startTime, time_t stopTime,
assert(startTime);
assert(stopTime);
txn.parameterized
("update BuildSteps set busy = 0, status = $1, propagatedFrom = $4, errorMsg = $5, startTime = $6, stopTime = $7, machine = $8, overhead = $9 where build = $2 and stepnr = $3")
("update build_steps set busy = 0, status = $1, propagated_from = $4, error_msg = $5, start_time = $6, stop_time = $7, machine = $8, overhead = $9 where build = $2 and stepnr = $3")
((int) status)(buildId)(stepNr)
(propagatedFrom, propagatedFrom != 0)
(errorMsg, errorMsg != "")
Expand All @@ -276,7 +276,7 @@ int State::createSubstitutionStep(pqxx::work & txn, time_t startTime, time_t sto
int stepNr = allocBuildStep(txn, build);

txn.parameterized
("insert into BuildSteps (build, stepnr, type, drvPath, busy, status, startTime, stopTime) values ($1, $2, $3, $4, $5, $6, $7, $8)")
("insert into build_steps (build, stepnr, type, drv_path, busy, status, start_time, stop_time) values ($1, $2, $3, $4, $5, $6, $7, $8)")
(build->id)
(stepNr)
(1) // == substitution
Expand All @@ -287,7 +287,7 @@ int State::createSubstitutionStep(pqxx::work & txn, time_t startTime, time_t sto
(stopTime).exec();

txn.parameterized
("insert into BuildStepOutputs (build, stepnr, name, path) values ($1, $2, $3, $4)")
("insert into build_step_outputs (build, stepnr, name, path) values ($1, $2, $3, $4)")
(build->id)(stepNr)(outputName)(storePath).exec();

return stepNr;
Expand Down Expand Up @@ -355,10 +355,10 @@ void State::markSucceededBuild(pqxx::work & txn, Build::ptr build,
{
if (build->finishedInDB) return;

if (txn.parameterized("select 1 from Builds where id = $1 and finished = 0")(build->id).exec().empty()) return;
if (txn.parameterized("select 1 from builds where id = $1 and finished = 0")(build->id).exec().empty()) return;

txn.parameterized
("update Builds set finished = 1, buildStatus = $2, startTime = $3, stopTime = $4, size = $5, closureSize = $6, releaseName = $7, isCachedBuild = $8 where id = $1")
("update builds set finished = 1, build_status = $2, start_time = $3, stop_time = $4, size = $5, closure_size = $6, release_name = $7, is_cached_build = $8 where id = $1")
(build->id)
((int) (res.failed ? bsFailedWithOutput : bsSuccess))
(startTime)
Expand All @@ -368,12 +368,12 @@ void State::markSucceededBuild(pqxx::work & txn, Build::ptr build,
(res.releaseName, res.releaseName != "")
(isCachedBuild ? 1 : 0).exec();

txn.parameterized("delete from BuildProducts where build = $1")(build->id).exec();
txn.parameterized("delete from build_products where build = $1")(build->id).exec();

unsigned int productNr = 1;
for (auto & product : res.products) {
txn.parameterized
("insert into BuildProducts (build, productnr, type, subtype, fileSize, sha1hash, sha256hash, path, name, defaultPath) values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)")
("insert into build_products (build, productnr, type, subtype, file_size, sha1hash, sha256hash, path, name, default_path) values ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)")
(build->id)
(productNr++)
(product.type)
Expand All @@ -386,11 +386,11 @@ void State::markSucceededBuild(pqxx::work & txn, Build::ptr build,
(product.defaultPath).exec();
}

txn.parameterized("delete from BuildMetrics where build = $1")(build->id).exec();
txn.parameterized("delete from build_metrics where build = $1")(build->id).exec();

for (auto & metric : res.metrics) {
txn.parameterized
("insert into BuildMetrics (build, name, unit, value, project, jobset, job, timestamp) values ($1, $2, $3, $4, $5, $6, $7, $8)")
("insert into build_metrics (build, name, unit, value, project, jobset, job, timestamp) values ($1, $2, $3, $4, $5, $6, $7, $8)")
(build->id)
(metric.second.name)
(metric.second.unit, metric.second.unit != "")
Expand All @@ -409,7 +409,7 @@ bool State::checkCachedFailure(Step::ptr step, Connection & conn)
{
pqxx::work txn(conn);
for (auto & path : step->drv.outputPaths())
if (!txn.parameterized("select 1 from FailedPaths where path = $1")(path).exec().empty())
if (!txn.parameterized("select 1 from failed_paths where path = $1")(path).exec().empty())
return true;
return false;
}
Expand Down Expand Up @@ -713,8 +713,8 @@ void State::dumpStatus(Connection & conn, bool log)
auto mc = startDbUpdate();
pqxx::work txn(conn);
// FIXME: use PostgreSQL 9.5 upsert.
txn.exec("delete from SystemStatus where what = 'queue-runner'");
txn.parameterized("insert into SystemStatus values ('queue-runner', $1)")(out.str()).exec();
txn.exec("delete from system_status where what = 'queue-runner'");
txn.parameterized("insert into system_status values ('queue-runner', $1)")(out.str()).exec();
txn.exec("notify status_dumped");
txn.commit();
}
Expand All @@ -732,7 +732,7 @@ void State::showStatus()
/* Get the last JSON status dump from the database. */
{
pqxx::work txn(*conn);
auto res = txn.exec("select status from SystemStatus where what = 'queue-runner'");
auto res = txn.exec("select status from system_status where what = 'queue-runner'");
if (res.size()) status = res[0][0].as<string>();
}

Expand All @@ -752,7 +752,7 @@ void State::showStatus()
/* Get the new status. */
{
pqxx::work txn(*conn);
auto res = txn.exec("select status from SystemStatus where what = 'queue-runner'");
auto res = txn.exec("select status from system_status where what = 'queue-runner'");
if (res.size()) status = res[0][0].as<string>();
}

Expand All @@ -779,7 +779,7 @@ void State::unlock()

{
pqxx::work txn(*conn);
txn.exec("delete from SystemStatus where what = 'queue-runner'");
txn.exec("delete from system_status where what = 'queue-runner'");
txn.commit();
}
}
Expand Down Expand Up @@ -862,7 +862,7 @@ void State::run(BuildID buildOne)
for (auto & step : steps) {
printMsg(lvlError, format("cleaning orphaned step %d of build %d") % step.second % step.first);
txn.parameterized
("update BuildSteps set busy = 0, status = $1 where build = $2 and stepnr = $3 and busy = 1")
("update build_steps set busy = 0, status = $1 where build = $2 and stepnr = $3 and busy = 1")
((int) bsAborted)
(step.first)
(step.second).exec();
Expand Down

0 comments on commit 28d10b4

Please sign in to comment.