Code repository for the IEEE Intelligent Vehicles Symposium 2019 paper on Dynamic Real-time Multimodal Routing (DREAMR) with Hierarchical Hybrid Planning (HHP). The paper is available on arXiv. Supplementary illustrative videos are available here and here.
N.B - The code is stable but is a continuous work in-progress. The experiments for the paper were done in Julia 0.6 and is available as the v0.2 release (the module is called HitchhikingDrones rather than DreamrHHP).
The DreamrHHP repository is set up as a package with its own environment in Julia 1.0. Look at Using someone else's project at the Julia package manager documentation for the basic idea. To get the code up and running, first install Julia. You then need to add the POMDPs.jl
registry; see Installation here. Once you have done the above, run the following:
$ git clone https://github.com/sisl/DreamrHHP.git
$ cd DreamrHHP
Then start the Julia REPL and go into package manager mode by pressing ]
, followed by:
(v1.0) pkg> activate .
(DreamrHHP) pkg> instantiate
This will install the necessary depedendencies and essentially reproduce the Julia environment required to make the package work.
This code is primarily provided for illustrative purposes. There are several moving parts that work with each other, rather than stand-alone. The description that follows assumes some familiarity with the paper.
The DreamrHHP
module (defined in the src
) folder implements the global open-loop layer, the local closed-loop layer, and any other utilities required to bridge them. The top-level HHP logic that invokes the two layers and interacts with the problem simulator, however, is in scripts/
as an executable Julia file but is not imported as such by the module.
The logic for the global layer is implemented in src/graph_plan/
. Following are the files of interest:
src/graph_plan/astar_visitor_light.jl
implements an implicit A* search, following the framework and convention of Graphs.jl. A re-implementation is necessary so that successors can be generated with a neighbour function, rather than using the edges of the graph.src/graph_plan/graph_solution.jl
implements the actual open-loop layer logic. It interfaces with the online route information that is available from the environment, updates the implicit graph, and calls the implicit A* search on it.src/graph_plan/graph_helpers.jl
implements a collection of auxiliary methods that are used byastar_visitor_light
, defines the vertex types and edge weight functions.
The logic required the local layer is implemented in src/macro_action_policy
and the code that actually generates the closed-loop policies for macro-actions is written in scripts/
. This layer's implementation relies heavily on the POMDPs.jl framework.
src/macro_action_policy/uavdynamics.jl
defines an interface for the agent dynamics (it can be any general dynamics model that an MDP can represent). It also implements a simple 2D multirotor model.src/macro_action_policy/partial_control_mdp.jl
defines the MDPs for constrained flight, unconstrained flight, and riding. All the components necessary to use LocalApproximationValueIteration to obtain policies for each of them, are also defined.scripts/pcmdp_hopon_*.jl
is a pair of scripts that generates constrained flight policies with varying values of the beta risk-threshold parameter for abort. From the paper, note that the abort logic can only be introduced in the CF policy after the partial control value iteration has been done once (to compute the worst values from each horizon). Therefore,_preabort
computes the value function for the CF MDP without abort logic, while_vary_abort_thresh
uses the in-horizon value function from_preabort
and then regenerates the in-horizon policy with a specific beta.scripts/ucmdp_
andscripts/unconstr_
execute logic similar toscripts/pcmdp_hopon_*
for the unconstrained flight and riding policies asscripts/pcmdp_hopon_
does. There is only one script for each because there is no varying parameter like beta for CF.
There are two components to the DREAMR simulation - the car routes which are updated online and the MDP simulator which implements the transition function and computes the trajectory cost. We separate the two in order to be able to generate car route data separately and more easily work with standalone route data if it should be available. For anyone wishing to benchmark, the code for simulators can be exported and used without any heavy dependencies.
data/grid_data_generator.jl
implements a generator for car routes for a full episode of 360 epochs, each of 5 seconds (total 30 minutes). It follows the format expected by the global open-loop layer logic insrc/graph_plan/graph_solution.jl
. The script generates a dictionary of information, with a nested dictionary for each epoch, and saves it as a JSON file.src/simulators/sdmc_simulator.jl
implements the transition and cost functions for the DREAMR MDP (sdmc is short-hand for Single Drone Multiple Cars). The car route information generated fromdata/grid_data_generator.jl
is provided to it (basically the car route info is generated standalone in advance), but the decision-making algorithm (HHP or RHC) only sees the updates to the currents information, and not the future. The general structure of theSDMCSimulator
resembles that of an OpenAI Gym environment.
The top-level code for HHP that invokes both layers and interacts with the DREAMR simulator is in scripts/sdmc_solver.jl
. It broadly matches the ONLINE
procedure in Algorithm 1 of the paper. It takes as arguments the offline policies, the parameter files, and the car routes for the full episode. The parameter and car route files are used to instantiate the SDMCSimulator
object, which the script then interacts with. The solver script can also log the drone state for each epoch during execution (in order to postprocess or visualize the solution).
The set of spatio-temporal and cost parameters define the specific policies and behaviour of the framework. There are two relevant components:
data/paramsets/*.toml
are the files that contain sets of values for the parameters in TOML format. We split the parameters intoscale-*.toml
for spatial parameters,simtime-*.toml
for temporal parameters, andcost-*.toml
for cost function parameters. This allows easy combinations of parameters for different settings. Thedata-allinone-*.toml
file is used bydata/grid_data_generator.jl
and contains parameters necessary for specifying the car routes.src/parameters.jl
implements aParameters
struct that reads in the parameter files and has nested structures calledScaleParameters
,SimTimeParameters
andCostParameters
respectively. The struct instantitions are passed around across the various code modules.