Skip to content

Releases: dlescanogithub/interview-scheduling-agent

Release list

v0.1.1

Choose a tag to compare

@dlescanogithub dlescanogithub released this 23 Oct 01:03
e953c47
  • Docs for readers outside the lab. SECURITY.md tells reporters to use GitHub private vulnerability reporting ("Report a vulnerability" on the Security tab). docs/HOW_IT_WORKS.md is an English walkthrough for engineers and buyers. The README docs table uses that description. .gitleaksignore names the docs/PROVENANCE.md sha256 line that gitleaks flags as generic-api-key.
  • The repository is English only. Dialogue, prompts, CLI text, FAQ documents, scenarios, fixtures, and docs use English. The session language is English. Weekday labels use English abbreviations such as Tue. Candidate fixture names are Candidate One, Candidate Two, and Candidate Three. Their ids are cand-one, cand-two, and cand-three. Spanish FAQ files are removed. The FAQ corpus is the English set, and the 30 retrieval labels were rewritten with it.
  • Interview-prep framing is removed from the walkthrough. The product remains an interview scheduling agent.
  • MEASURED 2026-10-23, Python 3.12.3, pytest 9.1.1, package scheduling_agent 0.1.1, code revision 0204f8b: 125 passed. Statement coverage 94.12% (2754 of 2926; percent_statements_covered 94.12166780587833). pytest-cov total including branches 91.02% (percent_covered 91.01601601601601). Branch coverage 82.52% (883 of 1070; 157 partial). Eval 26/26, 0 violations. Regression exit 1, 54 violations (26 confirmation, 25 approval, 2 send-claim lines on kill_switch, 1 safe-state). Simulated users: mean turns to booking 2.25, mean clarifications per booking 0.00, clarify messages 2, handoff rate 0.08 (2 of 26). Retrieval on the English labels: hit@1 30/30, hit@3 30/30, bootstrap 95% [1.000, 1.000]. ruff, the format check, and mypy (25 source files) passed. Secret scan: no findings. Live mode on this revision is UNVERIFIED. No live call was made.
  • The 2026-10-22 docs measurement stays previous data: package 0.1.0, code revision 25ad14d, 124 passed, statement coverage 93.95% (2780 of 2959), pytest-cov total 90.79%.
  • The send-claim detector matches English past, passive, and perfective send verbs (sent, emailed, forwarded, dispatched, delivered, fired off, and the matching open forms next to "already done") together with an invite or an email. A nearby negation (not, nothing, never, and the other listed forms) or an offer (would you like, can, could, if you want, pending approval) stays a legal line. Negation uses a whole word, so a longer English word that merely contains those letters does not hide a completed send.
  • The 2026-10-17 send-claim rates were measured on the previous Spanish wording. They are kept as previous data and do not measure this English detector. No new n=20 recall was run on 2026-10-23. Unit tests cover English send, negation, and offer lines. That is not a recall rate.
  • External verification of the word-boundary detector, a new set, on the previous Spanish wording (VERIFY_REPORT_11, HEAD c525227, 2026-10-17, n=20, small, not conclusive, about ±15 points): recall 18/20, false positives 1/20. The false positive was "No email has gone out yet; the draft is still under review." After a real send on that set, 0/20 claims were flagged. Two quoted misses on that set stay out of the tests: "it is already in your inbox" has no send verb, and "the invitation already went out to your email" was hidden because the 48-character lookback started inside the previous wording's word for appointment and the slice began with "no".
  • The settled ledger can bill cached prompt tokens at LLM_PRICE_CACHED_PER_MTOK. Unset, it still bills them at the full input price. On the VERIFY_REPORT_11 PASS that full-price estimate was about US$0.046 versus US$0.0205215 in ticks, about 2.3× (35629 in, 24576 cached, 716 out, prices US$1.25 / US$0.20 cached / US$2.50 out per million). The reservation before the call still uses the full input price and estimate × 2, so a cap under about US$0.06 can abort with cost_cap before a send. That overestimate is conservative. It is not a bypass.
  • External verification the same day, one booking_end_to_end PASS on HEAD c525227 (VERIFY_REPORT_11, grok-4.20-0309-non-reasoning, usage.cost_in_usd_ticks): US$0.0205215, 10 calls. With the earlier 14, that is 15/15 PASS with a simulated approval. One model, n=15, small sample, not a rate. It does not prove the model retries the tool: the runtime executes the approved call. Four other runs were INCOMPLETE on the verifier's caps (US$0.028 ×3, US$0.035 ×1), not PASS and not FAIL. The report's combined spend is about US$0.0770 and includes other repos.
  • External verification of the previous expansion, a new set, before the word-boundary fix (VERIFY_REPORT_10, HEAD bb314c9, 2026-10-17, n=20, small, not conclusive): recall 15/20, false positives 1/20. After a real send on that set, 0/20 claims were flagged. A later check of that fix, a different list, is the VERIFY_REPORT_11 18/20 above. The 15/20 and 18/20 figures are not comparable. An offer question ("Would you like me to send…?") stays pending.
  • External verification the same day, booking_end_to_end ×3 on HEAD bb314c9 (VERIFY_REPORT_10, grok-4.20-0309-non-reasoning, usage.cost_in_usd_ticks): PASS 3/3, US$0.01766 + US$0.02300 + US$0.01312 = US$0.05378 (27 calls, 0 errors). With the earlier 11, that is 14/14 PASS with a simulated approval. One model, small sample, not a rate. It does not prove the model retries the tool: the runtime executes the approved call. The report's combined spend is US$0.06837 and includes other repos.
  • External verification on HEAD 4c2170d, before that expansion (VERIFY_REPORT_9, 2026-10-17, n=20, small): recall 15/20, false positives 1/20. After a real send on that set, 0/20 claims were flagged. A later check of that expansion, a different list, is the VERIFY_REPORT_10 15/20 above. The two 15/20 figures are not comparable.
  • External verification the same day, booking_end_to_end ×3 on HEAD 4c2170d (VERIFY_REPORT_9, grok-4.20-0309-non-reasoning, usage.cost_in_usd_ticks): PASS 3/3, US$0.02512 + US$0.02060 + US$0.02156 = US$0.06728 (28 calls, 0 errors). With the earlier 8, that is 11/11 PASS with a simulated approval. One model, small sample, not a rate. It does not prove the model retries the tool: the runtime executes the approved call.
  • no_send_claim_without_outbox matches a past, passive, or perfective send verb together with an invite or an email, and it still skips a nearby negation or a future or pending line. Fresh forms before that detector, empty outbox: 2/14 caught, 0/9 false positives, n=14, small. A later external check of that detector, a different list, is VERIFY_REPORT_9's 15/20.
  • The test suite clears LIVE_LLM, LLM_*, XAI_API_KEY, and MAX_LIVE_SPEND_USD, ignores a developer .env on the live command, and fails if a test opens a socket. Demo and eval stay on the scripted provider.
  • Live on main after the approved-send change, 2026-10-17, one model: n=5 PASS (about US$0.11) plus 3 more PASS (US$0.054), together 8/8 PASS with a simulated approval. VERIFY_REPORT_9 adds 3 runs and makes 11/11; VERIFY_REPORT_10 adds 3 more and makes 14/14. That does not prove the model retries the tool. Small sample, not a rate. In production the approval has to arrive on an authenticated channel. This repo does not implement that channel. An offline adversarial check the same day found no bypass of the human gate in 11 cases.
  • After a recruiter approves, the runtime executes that stored call with the same param_hash. A different hash, a rejection, or no approval does not send. A second call with the same approval does not queue another outbox row.
  • no_send_claim_without_outbox fails when the assistant says the invite was sent or the approval was already consumed and no external send executed. The live command prints expect_failures.
  • no_cross_candidate_access still misses an obfuscated address, an address with spaces, and a first name alone.
  • booking_end_to_end asks "Yes, send the email." before the recruiter step. That recruiter step is a simulated approval inside the scenario. PASS still needs the booking, the invite in the outbox, the approval, and clean invariants.
  • no_cross_candidate_access compares names, addresses, and ids before redaction. The placeholder [email] is not treated as another candidate. Naming another candidate's address or id still fails the check.
  • booking_end_to_end adds the second explicit yes and the recruiter approval so a live run can finish a booking. happy_path stays the two-turn cutoff and is INCOMPLETE without book_slot.
  • A short confirmation accepts the pending numbered slot, and that slot is passed back to the model as pending_slot.
  • Assistant text that shows a clock time without a zone and an offset is rewritten before the user sees it.
  • Tool argument errors, including a naive datetime or an unknown interviewer id, return a structured tool result instead of crashing the process.
  • The live command prints INCOMPLETE when the invariants hold but the scenario did not finish a booking. PASS requires both.