Skip to content
Permalink
Browse files

Auto merge of #450 - ecstatic-morse:capabilities, r=pietroalbini

Add experiment requirements

Resolves #430.

This adds a `requirement` field to `Experiment`, which specifies what requirements (if any) an agent must meet to run that experiment.  Requirements are not structured, they are simply strings which are compared for equality. Agents advertise what requirements they meet via the request body when getting their configuration from the `/config` endpoint. Only experiments whose requirements are a subset of an agent's capabilities can be assigned to that agent. By default, experiments created via the CLI have no requirements--they can be run on any machine--and ones created via webhook (e.g. from `craterbot`) have the `linux`requirement--they must be run on a Linux machine.

For now, an experiment can have zero or one requirements. In the future, we may want to allow experiments to have an arbitrary number of requirements, and only agents which match all of them will be selected to run that experiment. This would require an additional table however.

For a crater run to be scheduled on both a Linux and Windows agent, you'll need to invoke `craterbot` twice, once with the `linux` requirement and once with the `windows` requirement. Perhaps later we could add some sugar for this?

r? @pietroalbini
  • Loading branch information...
bors committed Sep 9, 2019
2 parents 4d6cc3b + d54cd7f commit 1ed788cda06dbaaa867b1f4644cba9e089e41aa7
@@ -64,11 +64,28 @@ behave this way:
All the endpoints return a JSON response with a 200 status code if the request
succeded.

### `GET /config`
### `POST /config`

This endpoint returns the generic configuration of this agent, assigned by the
crater server. This method should be called at least at the start of the agent,
and the response is tied to the API token.
This endpoint registers the capabilities of this agent, and returns the generic
configuration of this agent, assigned by the crater server. This method should
be called at least at the start of the agent, and the response is tied to the
API token.

The agent's capabilities must be specified as JSON in the request body. Prior
to the introduction of capabilities, this endpoint was accessed via a `GET`
instead of a `POST`. A `GET` to this endpoint will still register an agent, but
the request body will be ignored and a default set of capabilities used instead
(currently `["linux"]`).

Request fields:

* `capabilities`: an array containing the capabilities possessed by this agent.

```json
{
"capabilities": ["windows", "hard-drive-bigger-than-1TB"]
}
```

Response fields:

@@ -88,10 +105,10 @@ Response fields:
### `GET /next-experiment`

This endpoint returns the next experiment this agent should run. The first time
this method is called the first queued experiment is assigned to the agent, and
its configuration is returned. The same configuration is returned for all the
following calls, until the agent sends the full experiment result to the crater
server.
this method is called the first queued experiment with compatible requirements
is assigned to the agent, and its configuration is returned. The same
configuration is returned for all the following calls, until the agent sends
the full experiment result to the crater server.

Response fields:

@@ -102,6 +102,15 @@ commands. At the moment, the name is predicted in these cases:

[Go back to the TOC][h-toc]

## Experiment requirements

Crater uses a system of requirements and capabilities to control which class of
agent can run which experiments. For now, there are two classes of agents:
Linux agents have the capability `linux`, and Windows agents have the
capability `windows`. You must specify a requirement for your experiment
(either `linux` or `windows`), and your experiment will only run on agents with
that capability.

## Commands reference

### Creating experiments
@@ -125,6 +134,7 @@ beta run you can use:
* `crates`: the selection of crates to use (default: `full`)
* `cap-lints`: the lints cap (default: `forbid`, which means no cap)
* `ignore-blacklist`: whether the blacklist should be ignored (default: `false`)
* `requirement`: any requirement of the agent running the experiment (default: `linux`)
* `assign`: assign the experiment to a specific agent (use this only when you
know what you're doing)
* `p`: the priority of the run (default: `0`)
@@ -153,6 +163,7 @@ priority of the `foo` experiment you can use:
* `crates`: the selection of crates to use (default: `full`)
* `cap-lints`: the lints cap (default: `forbid`, which means no cap)
* `ignore-blacklist`: whether the blacklist should be ignored (default: `false`)
* `requirement`: any requirement of the agent running the experiment (default: `linux`)
* `assign`: assign the experiment to a specific agent (use this only when you
know what you're doing)
* `p`: the priority of the run (default: `0`)
@@ -15,6 +15,7 @@ pub struct CreateExperiment {
pub github_issue: Option<GitHubIssue>,
pub ignore_blacklist: bool,
pub assign: Option<Assignee>,
pub requirement: Option<String>,
}

impl CreateExperiment {
@@ -32,6 +33,7 @@ impl CreateExperiment {
github_issue: None,
ignore_blacklist: false,
assign: None,
requirement: None,
}
}
}
@@ -55,8 +57,8 @@ impl Action for CreateExperiment {
"INSERT INTO experiments \
(name, mode, cap_lints, toolchain_start, toolchain_end, priority, created_at, \
status, github_issue, github_issue_url, github_issue_number, ignore_blacklist, \
assigned_to) \
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, ?12, ?13);",
assigned_to, requirement) \
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?11, ?12, ?13, ?14);",
&[
&self.name,
&self.mode.to_str(),
@@ -71,6 +73,7 @@ impl Action for CreateExperiment {
&self.github_issue.as_ref().map(|i| i.number),
&self.ignore_blacklist,
&self.assign.map(|a| a.to_string()),
&self.requirement,
],
)?;

@@ -126,6 +129,7 @@ mod tests {
}),
ignore_blacklist: true,
assign: None,
requirement: Some("linux".to_string()),
}
.apply(&ctx)
.unwrap();
@@ -152,6 +156,7 @@ mod tests {
assert_eq!(ex.status, Status::Queued);
assert!(ex.assigned_to.is_none());
assert!(ex.ignore_blacklist);
assert_eq!(ex.requirement, Some("linux".to_string()));
}

#[test]
@@ -250,6 +255,7 @@ mod tests {
github_issue: None,
ignore_blacklist: false,
assign: None,
requirement: None,
}
.apply(&ctx)
.unwrap_err();
@@ -279,6 +285,7 @@ mod tests {
github_issue: None,
ignore_blacklist: false,
assign: None,
requirement: None,
}
.apply(&ctx)
.unwrap();
@@ -294,6 +301,7 @@ mod tests {
github_issue: None,
ignore_blacklist: false,
assign: None,
requirement: None,
}
.apply(&ctx)
.unwrap_err();
@@ -13,6 +13,7 @@ pub struct EditExperiment {
pub priority: Option<i32>,
pub ignore_blacklist: Option<bool>,
pub assign: Option<Assignee>,
pub requirement: Option<String>,
}

impl EditExperiment {
@@ -27,6 +28,7 @@ impl EditExperiment {
priority: None,
ignore_blacklist: None,
assign: None,
requirement: None,
}
}
}
@@ -146,6 +148,16 @@ impl Action for EditExperiment {
ex.assigned_to = Some(assign);
}

// Try to update the requirement
if let Some(requirement) = self.requirement {
let changes = t.execute(
"UPDATE experiments SET requirement = ?1 WHERE name = ?2;",
&[&requirement.to_string(), &self.name],
)?;
assert_eq!(changes, 1);
ex.requirement = Some(requirement);
}

Ok(())
})?;
Ok(())
@@ -193,6 +205,7 @@ mod tests {
github_issue: None,
ignore_blacklist: false,
assign: None,
requirement: None,
}
.apply(&ctx)
.unwrap();
@@ -210,6 +223,7 @@ mod tests {
priority: Some(10),
ignore_blacklist: Some(true),
assign: Some(Assignee::CLI),
requirement: Some("windows".to_string()),
}
.apply(&ctx)
.unwrap();
@@ -224,6 +238,7 @@ mod tests {
assert_eq!(ex.priority, 10);
assert_eq!(ex.ignore_blacklist, true);
assert_eq!(ex.assigned_to, Some(Assignee::CLI));
assert_eq!(ex.requirement, Some("windows".to_string()));

assert_eq!(
ex.get_crates(&ctx.db).unwrap(),
@@ -1,3 +1,4 @@
use crate::agent::Capabilities;
use crate::crates::{Crate, GitHubRepo};
use crate::experiments::Experiment;
use crate::prelude::*;
@@ -120,9 +121,10 @@ impl AgentApi {
}
}

pub fn config(&self) -> Fallible<AgentConfig> {
pub fn config(&self, caps: &Capabilities) -> Fallible<AgentConfig> {
self.retry(|this| {
this.build_request(Method::GET, "config")
this.build_request(Method::POST, "config")
.json(&json!(caps))
.send()?
.to_api_response()
})
@@ -5,25 +5,76 @@ use crate::agent::api::AgentApi;
use crate::agent::results::ResultsUploader;
use crate::config::Config;
use crate::crates::Crate;
use crate::db::{Database, QueryUtils};
use crate::experiments::Experiment;
use crate::prelude::*;
use crate::utils;
use failure::Error;
use rustwide::Workspace;
use std::collections::BTreeSet;
use std::iter::FromIterator;
use std::ops;
use std::thread;
use std::time::Duration;

#[derive(Default, Serialize, Deserialize)]
pub struct Capabilities {
#[serde(default)]
capabilities: BTreeSet<String>,
}

impl ops::Deref for Capabilities {
type Target = BTreeSet<String>;

fn deref(&self) -> &Self::Target {
&self.capabilities
}
}

impl ops::DerefMut for Capabilities {
fn deref_mut(&mut self) -> &mut Self::Target {
&mut self.capabilities
}
}

impl Capabilities {
pub fn new(caps: &[&str]) -> Self {
let capabilities = caps.iter().map(|s| s.to_string()).collect();
Capabilities { capabilities }
}

pub fn for_agent(db: &Database, agent: &str) -> Fallible<Self> {
let caps = db.query(
"SELECT capability FROM agent_capabilities WHERE agent_name = ?1",
&[&agent],
|r| r.get("capability"),
)?;

Ok(caps.into_iter().collect())
}
}

impl FromIterator<String> for Capabilities {
fn from_iter<T>(iter: T) -> Self
where
T: IntoIterator<Item = String>,
{
let capabilities = iter.into_iter().collect();
Capabilities { capabilities }
}
}

struct Agent {
api: AgentApi,
config: Config,
}

impl Agent {
fn new(url: &str, token: &str) -> Fallible<Self> {
fn new(url: &str, token: &str, caps: &Capabilities) -> Fallible<Self> {
info!("connecting to crater server {}...", url);

let api = AgentApi::new(url, token);
let config = api.config()?;
let config = api.config(caps)?;

info!("connected to the crater server!");
info!("assigned agent name: {}", config.agent_name);
@@ -63,8 +114,14 @@ fn run_experiment(
Ok(())
}

pub fn run(url: &str, token: &str, threads_count: usize, workspace: &Workspace) -> Fallible<()> {
let agent = Agent::new(url, token)?;
pub fn run(
url: &str,
token: &str,
threads_count: usize,
caps: &Capabilities,
workspace: &Workspace,
) -> Fallible<()> {
let agent = Agent::new(url, token, caps)?;
let db = results::ResultsUploader::new(&agent.api);

run_heartbeat(url, token);

0 comments on commit 1ed788c

Please sign in to comment.
You can’t perform that action at this time.