feat: raise the claim and run the job again after a kill for memory

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "qex"
-version = "0.7.1"
+version = "0.8.1"
 edition = "2021"
 description = "Queued EXecutor — a resource-aware local job queue for long-running tasks"
 license = "Apache-2.0"

README.md

@@ -149,6 +149,36 @@ coordinated.
 which limits the damage, but an accurate claim is better. qex measures each job
 and uses the measurement for the next job of the same command.
 
+## When the kernel stops a job for memory
+
+A training run with `--mem guess` that the kernel stops at hour four gets the
+state `oom`. That state says one thing: **the claim was too small**.
+
+qex corrects it. It multiplies the claim, starts the job again with the same id
+and the same record, and says in the record what it did:
+
+```
+claim:     1 core(s), 16GB  (qex raised it, because the earlier claim was too small)
+note:      the kernel stopped attempt 1 of this job, because the job used more
+           memory than its claim of 8GB. THE CLAIM WAS TOO SMALL. qex raised
+           the claim to 16GB and starts the job again.
+```
+
+The job goes through the queue again, so its new claim meets the budget in the
+same way as a new job. qex also keeps the lesson: that kill says that the
+command needs **more than** 8GB, so the next run of the same command starts
+above that value and does not die in the same way.
+
+qex makes this correction when it applied the memory limit itself, with
+`[enforce] mode`. The kernel then stopped the job at the claim, and the kill is
+proof. With no limit — the default — qex can read the count of the login session
+only, and that count also rises when the kernel stops a different program of the
+same user. qex then reports the state `oom`, says what you can do, and starts no
+new attempt: the machine can be full while your claim is correct.
+
+See [the reference](docs/reference.md#a-job-that-the-kernel-stops-for-memory)
+for the limit on the raises.
+
 ## The documentation
 
 The full documentation is at

docs/design.md

@@ -177,9 +177,18 @@ This is deliberate, and the reason is the job that climbs for hours: a claim
 that is too small stops the job, and a claim that is a little large costs
 capacity only.
 
-qex records a job that COMPLETED only. A job that somebody stopped, or that the
-out-of-memory killer stopped, shows the memory that it reached and not the
-memory that it needs, and that number would teach qex the wrong size.
+qex records a job that COMPLETED, and a job that the KERNEL STOPPED AT THE LIMIT
+THAT QEX APPLIED. The second kind is a lower bound and not a peak: the job did
+not finish, so the true need is above the value. qex records that bound when it
+made the cgroup of the job and read the counter of that cgroup; the counter of
+the login session also counts a kill in a different program of the same user,
+and the machine can be full while the claim of the job is correct. qex keeps the two kinds apart in `usage.json`, and
+the next claim goes above the largest of both. A lower bound costs a whole run
+to obtain, and an average with the smaller runs would lose it.
+
+qex records nothing else. A job that somebody stopped, or that reached its time
+limit, shows the memory that it reached and not the memory that it needs, and
+that number would teach qex the wrong size.
 
 ### Is there fairness between agents?
 

docs/reference.md

@@ -75,11 +75,88 @@ qex submit -- cargo test    # run 2: the claim comes from run 1
 the name, because `cargo build` and `cargo test` need different sizes. qex uses
 the **largest** measurement it holds plus a margin, because a claim that is too
 small stops the job while a claim that is a little large costs only capacity.
-A job that did not complete is never recorded: it shows the memory it reached,
-not the memory it needs.
+
+qex records two kinds of measurement, and it keeps them apart:
+
+| The job | What the sample says |
+| ------- | -------------------- |
+| completed | The memory that the job needs. |
+| the kernel stopped it at its own limit | A **lower bound**. The need is above this value. |
+
+qex keeps one lower bound for a command, because a ladder of attempts makes one
+at each step and they would fill the store. The learned claim also never goes
+above `[budget] mem`: qex makes that number itself, and it must not make a
+number that it then refuses.
+
+A lower bound costs a whole run to obtain, so qex never averages it away. The
+next claim is above it, and a smaller run that succeeds later does not remove
+it. A job that you stopped, or that reached its time limit, is never recorded:
+it shows the memory it reached, not the memory it needs.
 
 Turn it off with `[learn] enabled = false`.
 
+### A job that the kernel stops for memory
+
+The kernel stops a job that uses more memory than its claim, and the job gets
+the state `oom`. That state says one thing: **the claim was too small**.
+
+qex corrects it. It multiplies the claim, starts the job again with the same id
+and the same record, and writes in the record what it did:
+
+```
+$ qex status $ID
+state:     completed
+claim:     1 core(s), 16GB  (qex raised it, because the earlier claim was too small)
+note:      the kernel stopped attempt 1 of this job, because the job used more
+           memory than its claim of 8GB. THE CLAIM WAS TOO SMALL. qex raised
+           the claim to 16GB and starts the job again.
+attempts:  2
+```
+
+| Rule | Value |
+| ---- | ----- |
+| How many raises | `[retry] on_oom`, 2 by default. |
+| The multiplier | `[retry] growth`, 2.0 by default. |
+| The limit | The claim never goes above `[budget] mem`. |
+| The count | Separate from `--retries`, which stays for your own faults. |
+
+Each attempt costs the full time of the job, which is why the ladder has a
+limit. A job that stops for memory at the limit keeps the state `oom`, and the
+record tells you to give a larger `--mem` value or to use a larger machine.
+
+The job goes through the **queue** again. Its claim is now larger, and the queue
+never admitted that claim, so qex tests it against the budget in the same way as
+a new job. A raised job thus waits while other jobs hold the budget, and the sum
+of the claims stays inside the budget.
+
+#### When qex acts, and when it only reports
+
+qex finds a kill for memory with the count in `memory.events`, which Linux keeps
+for each cgroup. The counter of a cgroup counts the kills in each cgroup below
+it, so **where qex reads it decides what qex may do**:
+
+| `[enforce] mode` | What qex reads | What qex does |
+| ---------------- | -------------- | ------------- |
+| `soft` or `hard` | The cgroup that qex made for this job. | Reports `oom`, raises the claim, runs the job again, and teaches the learner. |
+| `off` (default)  | The cgroup of your login session. | Reports `oom` and says what you can do. It starts no new attempt and teaches the learner nothing. |
+
+With no limit, the count also rises when the kernel stops a **different program
+of the same user**. A machine that is short of memory is also the machine on
+which a person uses `kill -9`, so the two events arrive together. That evidence
+does not prove that the claim of this job was too small: the machine can be full
+while the claim is correct. qex therefore reports the state and stops.
+
+To get the correction, set `[enforce] mode`. The kernel then stops the job at
+the claim, and a kill is proof that the claim was too small.
+
+`qex kill` writes a mark before it sends the signal, and that mark always wins.
+A job that you stopped never runs again with a larger claim, and it teaches the
+learner nothing.
+
+On a machine with no cgroup, such as macOS, qex has no count. A `SIGKILL` that
+no qex command sent then gives the state `killed`, which starts no new attempt,
+and the record says that qex could not tell the cause.
+
 Do not run a small test job to measure a task. Give `guess` and start the real
 task. qex measures each job, and you can read the true use later:
 
@@ -234,6 +311,10 @@ max_pressure = 20     # maximum PSI memory pressure (Linux only)
 [queue]
 oversized = "run-when-idle"   # run-when-idle, reject or queue
 
+[retry]
+on_oom = 2            # times to raise the claim after a kill for memory
+growth = 2.0          # the multiplier for the claim at each raise
+
 [defaults]
 cpu = 1               # the default is 1 core
 mem = "2GB"           # the default is the machine memory / the core count

src/commands.rs

@@ -385,6 +385,9 @@ fn print_status(s: &JobStatus, show_env: bool) -> Result<()> {
             // calculated it from the earlier jobs, and that no agent chose it.
             "learned" => "  (from the earlier jobs of this command)",
             "default" => "  (the default; give --cpu and --mem to change it)",
+            // A claim that qex raised must say so. Without this text, a reader
+            // sees a number that no agent gave and no measurement produced.
+            "raised" => "  (qex raised it, because the earlier claim was too small)",
             _ => "",
         }
     );
@@ -416,7 +419,15 @@ fn print_status(s: &JobStatus, show_env: bool) -> Result<()> {
         println!("waits for: {r}");
     }
     if let Some(e) = &s.error {
-        println!("error:     {e}");
+        // A job that succeeded can also hold a text here. qex writes the story
+        // of an out-of-memory kill in this field, and that story stays in the
+        // record after a later attempt succeeded. The word `error` would then
+        // contradict the state, so this line uses the word `note`.
+        if s.state == JobState::Completed {
+            println!("note:      {e}");
+        } else {
+            println!("error:     {e}");
+        }
     }
     if s.attempts > 1 || s.retries_left > 0 {
         println!(
@@ -1261,13 +1272,20 @@ fn describe_result(s: &JobStatus) -> String {
         },
         JobState::Killed => "a command stopped the job".to_string(),
         JobState::Timeout => "the job reached its time limit".to_string(),
-        JobState::Oom => {
+        // Say that the CLAIM was too small, and not that the machine was full.
+        //
+        // The words "the machine ran out of memory" sent the reader to the
+        // machine, and the fault was in the claim. qex holds the full story in
+        // the error field, with the claim that it tried, so give that text when
+        // qex wrote it.
+        JobState::Oom => s.error.clone().unwrap_or_else(|| {
             format!(
-                "the machine ran out of memory. The job claimed {} and used {}.",
+                "the kernel stopped the job for memory. The claim of {} was too small, and the \
+                 job reached {}. Give a larger `--mem` value.",
                 format_size(s.mem),
                 format_size(s.usage.max_rss)
             )
-        }
+        }),
         JobState::Cancelled => "the job left the queue".to_string(),
         // Give the cause here. A reader of the last job of a pipeline then
         // learns which job failed, with no other command.
@@ -1963,6 +1981,21 @@ pub fn rerun(args: cli::RerunArgs) -> Result<i32> {
     let mut spec = crate::job::read_spec(&dir)
         .with_context(|| format!("reading the specification of the job {id}"))?;
 
+    // Use the claim IN FORCE, and not the claim of the submission.
+    //
+    // The record holds the claim that the job had at the end. That value is the
+    // value of the specification, except after a kill for memory: qex then
+    // raised the claim, and the job succeeded at the larger value. A rerun from
+    // the specification would repeat the claim that the kernel already stopped,
+    // and the correction that cost a whole run would go away.
+    if let Ok(status) = crate::job::read_status(&dir) {
+        if status.mem > spec.mem {
+            spec.mem = status.mem;
+            spec.cpu = status.cpu.max(spec.cpu);
+            spec.claim_source = status.claim_source.clone();
+        }
+    }
+
     // A new job needs a new id, and it must not keep the dependencies of the
     // first job: those jobs have stopped, and a dependency on a job that
     // succeeded is not correct.
@@ -2404,6 +2437,7 @@ mod tests {
             locks: vec![],
             attempts: 1,
             retries_left: 0,
+            oom_raises: 0,
             caused_by: None,
             tags: vec![],
         }
@@ -2465,6 +2499,19 @@ mod tests {
         // The text must give the claim and the true use. An agent then corrects
         // its claim from this line.
         assert!(text.contains("1GB") && text.contains("2GB"), "got: {text}");
+        // It must also name the CLAIM as the fault. The words "the machine ran
+        // out of memory" sent the reader to the machine, and the fault was in
+        // the claim.
+        assert!(text.contains("too small"), "got: {text}");
+
+        // qex writes the full story in the error field: the claim that failed,
+        // the new claim, and the attempt. That text must win, because it says
+        // more than the line above.
+        s.error = Some("qex raised the claim to 2GB and starts the job again".into());
+        assert!(
+            describe_result(&s).contains("raised the claim"),
+            "the record of qex must win"
+        );
     }
 
     #[test]

src/config.rs

@@ -289,6 +289,39 @@ impl Default for LearnConfig {
     }
 }
 
+/// Controls what qex does when the kernel stops a job for memory.
+#[derive(Debug, Clone, Deserialize, Serialize)]
+#[serde(default, deny_unknown_fields)]
+pub struct RetryConfig {
+    /// The number of times that qex raises the claim and starts the job again.
+    ///
+    /// This count is SEPARATE from `--retries`, and the reason is important.
+    /// `--retries` is for a fault outside the task, such as a network that is
+    /// not ready, and the user chose that number for that fault. A kill for
+    /// memory is a fault of the CLAIM, and qex made the claim in the usual
+    /// case: `--mem guess` and the learned claim both come from qex. qex must
+    /// therefore correct its own fault, and it must not spend a budget that the
+    /// user gave for a different purpose. A job with no `--retries` value thus
+    /// still gets this correction.
+    ///
+    /// The count has a limit, and 2 raises give 4 times the first claim. Each
+    /// attempt costs the full time of the job: the job in the README runs for
+    /// four hours before the kernel stops it. A ladder with no limit can thus
+    /// use a day of the machine and give no result.
+    pub on_oom: u32,
+    /// The multiplier for the claim at each raise.
+    pub growth: f64,
+}
+
+impl Default for RetryConfig {
+    fn default() -> Self {
+        Self {
+            on_oom: 2,
+            growth: 2.0,
+        }
+    }
+}
+
 #[derive(Debug, Clone, Default, Deserialize, Serialize)]
 #[serde(default, deny_unknown_fields)]
 pub struct Config {
@@ -300,6 +333,7 @@ pub struct Config {
     pub submit: SubmitConfig,
     pub defaults: DefaultsConfig,
     pub learn: LearnConfig,
+    pub retry: RetryConfig,
     pub history: HistoryConfig,
     pub gc: GcConfig,
 }
@@ -430,6 +464,13 @@ impl Config {
                 self.learn.margin
             );
         }
+        if self.retry.on_oom > 0 && self.retry.growth <= 1.0 {
+            anyhow::bail!(
+                "config [retry] growth is {}. Use a value above 1.0. A smaller value gives the \
+                 claim that the kernel already stopped, and the job would stop again.",
+                self.retry.growth
+            );
+        }
         if self.enforce.mem_overcommit < 1.0 {
             anyhow::bail!(
                 "config [enforce] mem_overcommit is {}. Use a value of 1.0 or more. \
@@ -519,6 +560,10 @@ stale_after = "30s"
 oversized = "run-when-idle"
 settle = "3s"
 
+[retry]
+on_oom = 2
+growth = 2.0
+
 [submit]
 env_capture = "minimal"
 minimal_env = ["PATH", "HOME"]
@@ -531,6 +576,7 @@ timeout = "0"
         let c: Config = toml::from_str(text).unwrap();
         c.validate().unwrap();
         assert_eq!(c.enforce.mode, EnforceMode::Soft);
+        assert_eq!(c.retry.on_oom, 2);
         assert_eq!(c.submit.env_capture, EnvCapture::Minimal);
         assert_eq!(c.budget_mem().unwrap(), 20 << 30);
         assert_eq!(c.default_timeout().unwrap(), None);