The tag is the central organizing piece of data in HTMap. Every map that you run produces a Map which is connected to a unique tag. A tag cannot be re-used until the associated map has been deleted or retagged. You can either provide a tag or let HTMap generate one automatically.
If you do not provide a tag, the map will be marked as transient. Transient maps will be removed by the htmap.clean function without passing all = True, while non-transient (i.e., persistent) maps will not. If you provide a tag during map creation or htmap.Map.retag a map, it will be marked as persistent.
htmap.map
htmap.starmap
htmap.build_map
htmap.MapBuilder
__call__
__len__
A more convenient and flexible way to work with HTMap is to use the htmap decorator to build a MappedFunction.
htmap.mapped
htmap.MappedFunction
The Map is your window into the status and output of your map. Once you get a map result back from a map call, you can use its methods to get the status of jobs, change the properties of the map while its running, pause, restart, or cancel the map, and finally retrieve the output once the map is done.
The various methods that allow you to get and iterate over components will raise exceptions if something has gone wrong with your map:
htmap.exceptions.MapComponentErrorif a component experienced an error while executing.htmap.exceptions.MapComponentHeldif a component was held by HTCondor, likely because an input file did not exist or the component used too much memory or disk.
The exception message will contain information about what caused the error. See error_handling for more details on error handling.
htmap.Map
__len__
__getitem__
htmap.ComponentStatus
htmap.MapStdOut
htmap.MapStdErr
htmap.MapOutputFiles
Map components can generally encounter two kinds of errors:
- An exception occurred inside your function on the execute node.
- HTCondor was unable to run the map component for some reason.
The first kind will result in HTMap transporting a htmap.ComponentError back to you, which you can access via htmap.Map.get_err. The htmap.ComponentError.report() method returns a formatted error report for your perusal. htmap.Map.error_reports is a shortcut that returns all of the error reports for all of the components of your map. If you want to access the error programmatically, you can grab it using htmap.get_err.
The second kind of error doesn't provide as much information. The method htmap.Map.holds will give you a dictionary mapping components to their htmap.ComponentHold, if they have one. htmap.Map.hold_report will return a formatted table showing any holds in your map. The hold's reason attribute will tell you a lot about what HTCondor doesn't like about your component.
htmap.ComponentError
htmap.ComponentHold
Map options are the equivalent of HTCondor's submit descriptors. All HTCondor submit descriptors are valid map options except those reserved by HTMap for internal use (see below).
Fixed options are the most basic option. The entire map will used the fixed option. If you pass a single string as the value of a map option, it will become a fixed option.
Variadic options are options that are given individually to each component of a map. For example, each component of a map might need a different amount of memory. In that case you could pass a list to request_memory, with the same number of elements as the number of inputs to the map.
Inherited options are given to a htmap.MappedFunction when it is created. Any maps made using that function can inherit these options. Options that are passed in the actual map call override inherited options (excepting fixed_input_files, see the note). For example, if you know that a certain function always takes a large amount of memory, you could give it a large request_memory at the htmap.MappedFunction level so that you don't have to do it for every individual map. Additionally, default map options can be set globally via settings['MAP_OPTIONS.<option_name>'] = <option_value>.
Warning
Only certain options make sense as inherited options. For example, they shouldn't be variadic options.
fixed_input_files has special behavior as an inherited option: they are merged together instead of overridden.
Note
When looking at examples of raw HTCondor submit files, you may see submit descriptors that are prefixed with a + or a MY.. Those options should be passed to htmap.MapOptions via the custom_options keyword arguments.
htmap.MapOptions
htmap.TransferPath
htmap.transfer_output_files
htmap.checkpoint
These functions help you manage your maps.
htmap.status
htmap.get_tags
htmap.load
htmap.load_maps
htmap.remove
htmap.clean
These functions are useful for generating machine-readable status information.
htmap.status_json
htmap.status_csv
htmap.register_delivery_method
These functions help you manage your transplant installs.
htmap.transplants
htmap.Transplant
htmap.transplant_info
HTMap exposes configurable settings through htmap.settings, which is an instance of the class htmap.settings.Settings. This settings object manages a lookup chain of dictionaries. The settings object created during startup contains two dictionaries. The lowest level contains HTMap's default settings, and the next level up is constructed from a settings file at ~/.htmaprc. If that file does not exist, an empty dictionary is used instead. The file should be formatted in TOML.
Alternate settings could be stored in other files or constructed at runtime. HTMap provides tools for saving, loading, merging, prepending, and appending settings to each other. Each map is search in order, so earlier settings can flexibly override later settings.
Warning
To entirely replace your settings, do not do htmap.settings = <new settings object>. Instead, use the htmap.settings.Settings.replace method. Replacing the settings by assignment breaks the internal linking between the settings objects and its dependencies.
Hint
These may be helpful when constructing fresh settings:
- HTMap's base settings are available as
htmap.BASE_SETTINGS. - The settings loaded from
~/.htmaprcare available ashtmap.USER_SETTINGS.
htmap.settings.Settings
HTMap exposes a standard Python logging hierarchy under the logger named 'htmap'. Logging configuration can be done by any of the methods described in the documentation.
Here's an example of how to set up basic console logging:
import logging
import sys
logger = logging.getLogger("htmap")
logger.setLevel(logging.DEBUG)
handler = logging.StreamHandler(stream=sys.stdout)
handler.setLevel(logging.DEBUG)
handler.setFormatter(
logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
)
logger.addHandler(handler)After executing this code, you should be able to see HTMap log messages as you tell it to do things.
Warning
The HTMap logger is not available in the context of the executing map function. Trying to use it will probably raise exceptions.
htmap.exceptions
htmap.version
htmap.version_info