Make parsing a dedicated experiment step. (#117)

docs/downward.tutorial.rst

@@ -123,15 +123,15 @@ Run tutorial experiment
 
 The files below are two experiment scripts, a ``project.py`` module that bundles
 common functionality for all experiments related to the project, a parser
-script, and a script for collecting results and making reports. You can use the
+module, and a script for collecting results and making reports. You can use the
 files as a basis for your own experiments. They are available in the `Lab repo
 <https://github.com/aibasel/lab/tree/main/examples/downward>`_. Copy the files
 into ``experiments/my-exp-dir``.
 
 .. highlight:: bash
 
-Make sure the experiment script and the parser are executable. Then you
-can see the available steps with ::
+Make sure the experiment script is executable. Then you can see the available
+steps with ::
 
     ./2020-09-11-A-cg-vs-ff.py
 
@@ -171,7 +171,7 @@ reference on all Downward Lab classes.
 .. literalinclude:: ../examples/downward/project.py
    :caption:
 
-.. literalinclude:: ../examples/downward/parser.py
+.. literalinclude:: ../examples/downward/custom_parser.py
    :caption:
 
 .. literalinclude:: ../examples/downward/01-evaluation.py

docs/faq.rst

@@ -38,12 +38,43 @@ again as above.
 I forgot to parse something. How can I run only the parsers again?
 ------------------------------------------------------------------
 
-See the `parsing documentation <lab.parser.html>`_ for how to write
-parsers. Once you have fixed your existing parsers or added new parsers,
-add ``exp.add_parse_again_step()`` to your experiment script
-``my-exp.py`` and then call ::
-
-    ./my-exp.py parse-again
+Now that parsing is done in its own experiment step, simply consult the `parsing
+documentation <lab.parser.html>`_ for how to amend your parsers and then run the
+"parse" experiment step again with ::
+
+    ./my-exp.py parse
+
+
+.. _portparsers:
+
+How do I port my parsers to version 8.x?
+----------------------------------------
+
+Since version 8.0, Lab has a dedicated "parse" experiment step. First of all,
+what are the benefits of this?
+
+* No need to write parsers in separate files.
+* Log output from solvers and parsers remains separate.
+* No need for ``exp.add_parse_again_step()``. Parsing and re-parsing is now
+  exactly the same.
+* Parsers are checked for syntax errors before the experiment is run.
+* Parsing runs much faster (for an experiment with 3 algorithms and 5 parsers
+  the parsing time went down from 51 minutes to 5 minutes, both measured on
+  cold file system caches).
+* As before, you can let the Slurm environment do the parsing for you and get
+  notified when the report is finished: ``./myexp.py build start parse fetch
+  report``
+
+To adapt your parsers to this new API, you need to make the following changes:
+
+* Your parser module (e.g., "custom_parser.py") does not have to be executable
+  anymore, but it must be importable and expose a :class:`Parser
+  <lab.parser.Parser>` instance (see the changes to the `translator parser
+  <https://github.com/aibasel/lab/pull/117/files#diff-0a679939eb576c6b402a00ab9b08a3339ecefe3713dc96f9ac6b0e05de9ff4f2>`_
+  for an example). Then, instead of ``exp.add_parser("custom_parser.py")`` use
+  ``from custom_parser import MyParser`` and ``exp.add_parser(MyParser())``.
+* Remove ``exp.add_parse_again_step()`` and insert ``exp.add_step("parse",
+  exp.parse)`` after ``exp.add_step("start", exp.start_runs)``.
 
 
 How can I compute a new attribute from multiple runs?

docs/ff.rst

@@ -27,5 +27,5 @@ Downward experiments, we recommend taking a look at the
 
 Here is a simple parser for FF:
 
-.. literalinclude:: ../examples/ff/ff-parser.py
+.. literalinclude:: ../examples/ff/ff_parser.py
    :caption:

docs/lab.concepts.rst

@@ -4,12 +4,13 @@ Concepts
 ========
 
 An **experiment** consists of multiple **steps**. Most experiments will
-have steps for building and executing the experiment:
+have steps for building and executing the experiment, and parsing logs:
 
     >>> from lab.experiment import Experiment
     >>> exp = Experiment()
     >>> exp.add_step("build", exp.build)
     >>> exp.add_step("start", exp.start_runs)
+    >>> exp.add_step("parse", exp.parse)
 
 Moreover, there are usually steps for **fetching** the results and making
 **reports**:
@@ -18,11 +19,11 @@ Moreover, there are usually steps for **fetching** the results and making
 >>> exp.add_fetcher(name="fetch")
 >>> exp.add_report(Report(attributes=["error"]))
 
-The "build" step creates all necessary files for running the experiment in
-the **experiment directory**. After the "start" step has finished running
-the experiment, we can fetch the result from the experiment directory to
-the **evaluation directory**. All reports only operate on evaluation
-directories.
+The "build" step creates all necessary files for running the experiment in the
+**experiment directory**. After the "start" step has finished running the
+experiment, we can parse data from logs and generated files into "properties"
+files, and then fetch all properties files from the experiment directory to the
+**evaluation directory**. All reports only operate on evaluation directories.
 
 An experiment usually also has multiple **runs**, one for each pair of
 algorithm and benchmark.

docs/lab.tutorial.rst

@@ -35,12 +35,7 @@ Select steps by name or index::
 
     ./exp.py build
     ./exp.py 2
-    ./exp.py 3 4
-
-Here is the parser that the experiment uses:
-
-.. literalinclude:: ../examples/vertex-cover/parser.py
-   :caption:
+    ./exp.py 3 4 5
 
 Find out how to create your own experiments by browsing the `Lab API
 <lab.experiment.html>`_.

docs/news.rst

@@ -1,6 +1,18 @@
 Changelog
 =========
 
+v8.0 (2023-10-21)
+-----------------
+
+Lab
+^^^
+* Make parsing a separate experiment step, see :ref:`FAQs <portparsers>` for motivation and upgrade instructions (Jendrik Seipp).
+
+Downward Lab
+^^^^^^^^^^^^
+* None.
+
+
 v7.5 (2023-10-21)
 -----------------
 

docs/singularity.rst

@@ -11,7 +11,7 @@ Downward Lab.
 
 The experiment script needs a parser and a helper script:
 
-.. literalinclude:: ../examples/singularity/singularity-parser.py
+.. literalinclude:: ../examples/singularity/singularity_parser.py
    :caption:
 
 .. literalinclude:: ../examples/singularity/run-singularity.sh

downward/experiment.py

@@ -9,14 +9,15 @@
 
 from downward import suites
 from downward.cached_revision import CachedFastDownwardRevision
+from downward.parsers.anytime_search_parser import AnytimeSearchParser
+from downward.parsers.exitcode_parser import ExitcodeParser
+from downward.parsers.planner_parser import PlannerParser
+from downward.parsers.single_search_parser import SingleSearchParser
+from downward.parsers.translator_parser import TranslatorParser
 from lab import tools
 from lab.experiment import Experiment, get_default_data_dir, Run
 
 
-DIR = os.path.dirname(os.path.abspath(__file__))
-DOWNWARD_SCRIPTS_DIR = os.path.join(DIR, "scripts")
-
-
 class FastDownwardAlgorithm:
     """
     A Fast Downward algorithm is the combination of revision, driver options and
@@ -122,32 +123,31 @@ class FastDownwardExperiment(Experiment):
         >>> exp = FastDownwardExperiment()
         >>> exp.add_step("build", exp.build)
         >>> exp.add_step("start", exp.start_runs)
+        >>> exp.add_step("parse", exp.parse)
         >>> exp.add_fetcher(name="fetch")
 
     """
 
     # Built-in parsers that can be passed to exp.add_parser().
 
     #: Parsed attributes: "error", "planner_exit_code", "unsolvable".
-    EXITCODE_PARSER = os.path.join(DOWNWARD_SCRIPTS_DIR, "exitcode-parser.py")
+    EXITCODE_PARSER = ExitcodeParser()
 
     #: Parsed attributes: "translator_peak_memory", "translator_time_done", etc.
-    TRANSLATOR_PARSER = os.path.join(DOWNWARD_SCRIPTS_DIR, "translator-parser.py")
+    TRANSLATOR_PARSER = TranslatorParser()
 
     #: Parsed attributes: "coverage", "memory", "total_time", etc.
-    SINGLE_SEARCH_PARSER = os.path.join(DOWNWARD_SCRIPTS_DIR, "single-search-parser.py")
+    SINGLE_SEARCH_PARSER = SingleSearchParser()
 
     #: Parsed attributes: "cost", "cost:all", "coverage".
-    ANYTIME_SEARCH_PARSER = os.path.join(
-        DOWNWARD_SCRIPTS_DIR, "anytime-search-parser.py"
-    )
+    ANYTIME_SEARCH_PARSER = AnytimeSearchParser()
 
     #: Used attributes: "memory", "total_time",
     #: "translator_peak_memory", "translator_time_done".
     #:
     #: Parsed attributes: "node", "planner_memory", "planner_time",
     #: "planner_wall_clock_time", "score_planner_memory", "score_planner_time".
-    PLANNER_PARSER = os.path.join(DOWNWARD_SCRIPTS_DIR, "planner-parser.py")
+    PLANNER_PARSER = PlannerParser()
 
     def __init__(self, path=None, environment=None, revision_cache=None):
         """

downward/scripts/anytime-search-parser.py → downward/parsers/anytime_search_parser.py

@@ -1,5 +1,3 @@
-#! /usr/bin/env python
-
 """
 Parse anytime-search runs of Fast Downward. This includes iterated
 searches and portfolios.
@@ -54,18 +52,18 @@ def add_memory(content, props):
             props["memory"] = raw_memory
 
 
-def main():
-    parser = Parser()
-    parser.add_pattern("raw_memory", r"Peak memory: (.+) KB", type=int),
-    parser.add_function(find_all_matches("cost:all", r"Plan cost: (.+)\n", type=float))
-    parser.add_function(
-        find_all_matches("steps:all", r"Plan length: (.+) step\(s\).\n", type=float)
-    )
-    parser.add_function(reduce_to_min("cost:all", "cost"))
-    parser.add_function(reduce_to_min("steps:all", "steps"))
-    parser.add_function(coverage)
-    parser.add_function(add_memory)
-    parser.parse()
-
-
-main()
+class AnytimeSearchParser(Parser):
+    def __init__(self):
+        super().__init__()
+
+        self.add_pattern("raw_memory", r"Peak memory: (.+) KB", type=int)
+        self.add_function(
+            find_all_matches("cost:all", r"Plan cost: (.+)\n", type=float)
+        )
+        self.add_function(
+            find_all_matches("steps:all", r"Plan length: (.+) step\(s\).\n", type=float)
+        )
+        self.add_function(reduce_to_min("cost:all", "cost"))
+        self.add_function(reduce_to_min("steps:all", "steps"))
+        self.add_function(coverage)
+        self.add_function(add_memory)

downward/scripts/exitcode-parser.py → downward/parsers/exitcode_parser.py

@@ -1,5 +1,3 @@
-#! /usr/bin/env python
-
 """
 Parse Fast Downward exit code and store a message describing the outcome
 in the "error" attribute.
@@ -43,9 +41,9 @@ def parse_exit_code(content, props):
         props.add_unexplained_error(outcome.msg)
 
 
-class ExitCodeParser(Parser):
+class ExitcodeParser(Parser):
     def __init__(self):
-        Parser.__init__(self)
+        super().__init__()
         self.add_pattern(
             "planner_exit_code",
             r"planner exit code: (.+)\n",
@@ -54,11 +52,3 @@ def __init__(self):
             required=True,
         )
         self.add_function(parse_exit_code)
-
-
-def main():
-    parser = ExitCodeParser()
-    parser.parse()
-
-
-main()

downward/scripts/planner-parser.py → downward/parsers/planner_parser.py

@@ -1,5 +1,3 @@
-#! /usr/bin/env python
-
 from lab import tools
 from lab.parser import Parser
 
@@ -96,11 +94,3 @@ def __init__(self):
         self.add_function(add_planner_memory)
         self.add_function(add_planner_time)
         self.add_function(add_planner_scores)
-
-
-def main():
-    parser = PlannerParser()
-    parser.parse()
-
-
-main()

downward/scripts/single-search-parser.py → downward/parsers/single_search_parser.py

@@ -1,5 +1,3 @@
-#! /usr/bin/env python
-
 """
 Regular expressions and functions for parsing single-search runs of Fast Downward.
 """
@@ -160,11 +158,3 @@ def __init__(self):
         self.add_function(add_initial_h_values)
         self.add_function(ensure_minimum_times)
         self.add_function(add_scores)
-
-
-def main():
-    parser = SingleSearchParser()
-    parser.parse()
-
-
-main()

downward/scripts/translator-parser.py → downward/parsers/translator_parser.py

@@ -1,5 +1,3 @@
-#! /usr/bin/env python
-
 """
 Regular expressions and functions for parsing translator logs.
 """
@@ -72,8 +70,3 @@ def __init__(self):
         self.add_function(parse_translator_timestamps)
         self.add_function(parse_old_statistics)
         self.add_function(parse_statistics)
-
-
-if __name__ == "__main__":
-    parser = TranslatorParser()
-    parser.parse()

examples/downward/2020-09-11-A-cg-vs-ff.py

@@ -3,6 +3,8 @@
 import os
 import shutil
 
+import custom_parser
+
 import project
 
 
@@ -64,11 +66,12 @@
 exp.add_parser(exp.EXITCODE_PARSER)
 exp.add_parser(exp.TRANSLATOR_PARSER)
 exp.add_parser(exp.SINGLE_SEARCH_PARSER)
-exp.add_parser(project.DIR / "parser.py")
+exp.add_parser(custom_parser.get_parser())
 exp.add_parser(exp.PLANNER_PARSER)
 
 exp.add_step("build", exp.build)
 exp.add_step("start", exp.start_runs)
+exp.add_step("parse", exp.parse)
 exp.add_fetcher(name="fetch")
 
 if not project.REMOTE:

examples/downward/2020-09-11-B-bounded-cost.py

@@ -4,6 +4,8 @@
 import os
 import shutil
 
+import custom_parser
+
 from downward import suites
 from downward.cached_revision import CachedFastDownwardRevision
 from downward.experiment import FastDownwardAlgorithm, FastDownwardRun
@@ -89,11 +91,12 @@
 exp.add_parser(project.FastDownwardExperiment.EXITCODE_PARSER)
 exp.add_parser(project.FastDownwardExperiment.TRANSLATOR_PARSER)
 exp.add_parser(project.FastDownwardExperiment.SINGLE_SEARCH_PARSER)
-exp.add_parser(project.DIR / "parser.py")
+exp.add_parser(custom_parser.get_parser())
 exp.add_parser(project.FastDownwardExperiment.PLANNER_PARSER)
 
 exp.add_step("build", exp.build)
 exp.add_step("start", exp.start_runs)
+exp.add_step("parse", exp.parse)
 exp.add_fetcher(name="fetch")
 
 if not project.REMOTE: