Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
..
Failed to load latest commit information.
python
README.md

README.md

Martian stage code adapter

This readme describes the interface between mrp and the mrjob monitor process, as well as the interface between the job monitor and the stage code, which must be obeyed by any language-specific adapter.

To support a new language, one must implement the adapter interface, which mostly amounts to reading and writing the right files.

Stage adapter interface

Command line arguments

The arguments to the adapter shell process are whatever was specified in the MRO for the stage (e.g. a path to the stage code), followed by:

  1. The run type: either split, joint, or main (for chunks).
  2. The path to the stage metadata files (see below).
  3. The path to the stage files directory. This is the directory where data files are written, and should also be the working directory.
  4. The journal file prefix (see below).

Open file descriptors

In addition to standard output and standard error (standard in is null), the monitor will leave two file handles open: 3. The stage log file. Generally this contains timestamps for progress stage process. This should not be closed by the adapter. 4. The error reporting pipe. In the event of a fatal error, write the error message to this descriptor and then close the pipe. If the message is prefixed with "ASSERT:" then the message is treated as an assertion. After closing the pipe, the adapter process will likely be killed shortly thereafter.

Files

A metadata file named "name" is placed in the path [metadata path]/_<name>. To notify mrp of the existence of a new file for a split or join, create a file named [journal prefix].<run type>_<name>. For chunks, use simply [journal prefix].<name>.

The args file contains the json-serialized arguments to the stage or chunk. Other common files depend on the phase.

  • Split phases must output a chunk_defs file. This file should contain a json-serialized array of chunk definitions, each of which is a dictionary containing the per-chunk arguments, and possibly a key named __threads and __mem_gb to specify the reservation for that job.
  • Chunk phases create an outs file with the json-serialized dictionary of output values. If the stage splits, the args and outs are per-chunk. If it does not, they are for the stage overall.
  • Join phases, in addition to the stage args and outs files, may read the chunk_defs. Additionally, the outs files for each chunk are aggregated by mrp into an array in the chunk_outs file.

Any stage may wish to access the jobinfo file. In some cases it may be appropriate to add to it. Be sure, however, that the updates to that file do not corrupt it.

A stage may update its progress file at any time. If it updates the journal file for progress then mrp will read the progress file and print its contents to the main mrp log next time it checks. Note that if mrp is restarted or is overwritten by a newer progress message before mrp sees that message, the older message will not make it into the mrp log.

The adapter process should generally not catch any signals except possibly those generated by the process itself, such as SIGSEGV. Catching signals is the responsibility of the monitor process.

Monitor interface

The monitor process, mrjob, is maintained with martian. Maintainers of language adapters do not need to understand its contracts unless they are also planning to maintain the monitor itself. Note that, unlike the API between mrjob and the adapter child process, mrjob is tightly coupled to the mrp version it was built with, and there should be no assumptions about the stability of the interface between them.

The job monitor process has several duties. If monitoring is enabled in the jobinfo file, the monitor will periodically inspect the process for its memory usage and kill it if it exceeds its memory reservation. In addition, it ensures that the correct files are created and journaled to ensure correct progress for mrp. In particular:

  • The log file is created when the job starts. It gets a timestamp when the job starts and when it completes, and may get additional messages through the file descriptor passed to the adapter.
  • If the stage code writes to the errors pipe, the monitor will read up to 8kb and put that in the errors metadata file (or assert if the message begins with ASSERT:) to indicate stage failure. If the stage process returns a nonzero exit code but does not write to the errors pipe, the exit code is written into errors.
  • If the stage process exits with a return code of 0, the monitor will write a complete file.
  • When the job completes, the jobinfo file is updated with performance information including rusage.
  • While the job is running, the journal for the heartbeat file is periodically written to make sure that mrp knows the job is still running.
  • If the profile mode indicates that an external profiler should be attached to the stage code, the monitor process handles that, including ensuring the profiler is given the correct output paths.

The monitor process catches certain signals. If a signal is caught, the errors file is written and the stage code is killed.

The arguments to the monitor process are passed through to the adapter except for the monitor executable itself.