Skip to content

rc14

Latest

Choose a tag to compare

@sileod sileod released this 02 Oct 13:30
· 117 commits to main since this release

55 tasks in seven capability areas. reasoning_core.list_tasks() returns them, and reasoning_core.registry.AREAS groups them.

Roster changes since rc10 (50 → 55)

Added (12): inverse_math, process_inversion, systems_trace, attribute_grammar, controlled_code_execution, dynamic_programming, finite_automaton_execution, rule_switching, shift_reduce_parsing, combinatorics_formula, function_manipulation, logic_derivation.
Removed (7): code_execution (replaced by controlled_code_execution, which measures the same and generates about 15× faster), lean_candidate_compilation and lean_missing_line (replaced by inverse_math), program_synthesis (parked as a dev task), graph_dependencies, logic_formalization and logic_nli. Removed tasks can still be loaded with get_task.

Task changes

  • multistep_evidence_retrieval v2 lists premises in a single alphabetical order. Before, the proof's facts came first, so the answer usually began at index 0. The problems are unchanged, and each prompt still has exactly one answer.
  • constraint_satisfaction generates 3–5× faster, with identical outputs.
  • Generation fixes for belief_tracking, multistep_nli, regex_reasoning, sequential_induction, most_probable_evidence and parsing_derivation.

Library

  • PROMOTED in the registry can now add a dev task to the roster by name. Its file and identity stay the same.
  • reasoning_core.evaluation now holds the LLM judges, the cheap Jev answer-plausibility and difficulty signals, and the level audit. They are shared with task_search. Judge configuration uses the neutral RC_JUDGE_* environment variables.
  • The dataset card's task catalogue is generated from registry.AREAS.

Tasks by area

  • Logic & deduction (12): logic_qa, multistep_nli, multistep_abduction, multistep_evidence_retrieval, defeasible_nli, logic_derivation, unification_entailment, rewrite_system, lambda_reduction, metamath_core_select, metamath_entailment, analogical_case_matching
  • State tracking & execution (11): process_inversion, systems_trace, belief_tracking, reference_tracking, rule_switching, finite_automaton_execution, shift_reduce_parsing, controlled_code_execution, code_runnability, code_analysis, dynamic_programming
  • Mathematics (7): math_word_problem, inverse_math, combinatorics_formula, equation_system, arithmetics, function_manipulation, sequential_induction
  • Spatial, causal & probabilistic (7): planar_geometry_relations, constraint_satisfaction, qualitative_causal_reasoning, qualitative_reasoning, grid_navigation, most_probable_outcome, most_probable_evidence
  • Formal languages & parsing (7): attribute_grammar, regex_reasoning, regex_following, constrained_continuation, syntax_error_detection, parsing_derivation, string_transduction
  • Graphs, games & planning (6): game_forced_win, game_best_move, graph_pathfinding, graph_successors, planning, coreference
  • Sets & tables (5): set_expression, set_missing_element, table_qa, table_equivalence, table_statistics