Julia

Running Julia solvers from the GeoDMS

The GeoDMS exports a problem instance, a Julia program solves it, and the GeoDMS reads the solution back and maps it. The NetworkModel_EU configuration (branch ServiceAccess) is the reference implementation: it exports an origin-destination table with client and facility tables per country, solves the facility-location problem in lp.jl / greedy.jl / lambda_sweep_simplex.jl, and reads the resulting assignment and travel times back for mapping.

The mechanism is not Julia-specific - it is the same file exchange used for R and Python. What makes this configuration worth copying is that it is a production pipeline: many countries, several solver variants, and a driving batch file that keeps the steps in order.

the exchange format: Arrow

This pipeline uses Arrow rather than parquet. Both are written and read through GDAL with the same gdalwrite.vect / gdal.vect StorageManagers, and both carry the value type of every column in the file. Arrow is the natural choice here because Julia reads and writes it directly with Arrow.jl: Arrow.Table(path) to read, and Arrow.write(path, (id = ..., open = ...)) to write.

On the reading side the driver is named explicitly, because the .arrow extension is not recognised by every GDAL build:

parameter<string> GDAL_DRIVER := 'Arrow';

exporting the problem instance

Each export is an ordinary write storage; the StorageName is an indirect expression so that one configuration serves every study area:

container DistanceTableExport
:  StorageName     = "='%localDataProjDir%/'+ProjName+'/'+ModelParameters/StudyArea+'_od.arrow'"
,  StorageType     = "gdalwrite.vect"
,  StorageReadOnly = "False"
{
   attribute<Client>   Client_rel    (OD) := ../client_rel;
   attribute<Facility> facility_rel  (OD) := ../facility_rel;
   attribute<float32>  t_ij          (OD) := ../t_ij;
   attribute<float32>  d_ij          (OD) := ../d_ij;
   attribute<float32>  travelcost_ij (OD) := ../travelcost_ij;
   attribute<float32>  Deterrence_ij (OD) := ../Deterrence_ij;
}

The client and facility tables are exported the same way, and both export an explicit id attribute - attribute<client> id (client) := id(client);. Export that id. It is what lets the solver's output be joined back onto the configured domain unit later, and it costs one column.

driving the steps

Julia is not started from the configuration with exec_ec. The batch file owns the whole sequence and calls GeoDmsRun once per step, which is the only arrangement that orders a GeoDMS write correctly - see exec_ec for why a write cannot be forced to precede an exec_ec within one run.

set "GEODMS_EXE=C:\Program Files\ObjectVision\GeoDms20.16.0\GeoDmsRun.exe"
set "CFG=%~dp0cfg\main.dms"

set ITEMS_ALLOC="/Analyses/%ITEM_ANALYSIS%/Allocation/DistanceTableExport" "/Analyses/%ITEM_ANALYSIS%/Allocation/ClientExport" "/Analyses/%ITEM_ANALYSIS%/Allocation/FacilityExport"

for %%C in (%COUNTRIES%) do (
    set "STUDY_AREA=%%C"
    "%GEODMS_EXE%" /L"%LOG_DIR%\%%C_alloc.log" "%CFG%" %ITEMS_ALLOC%
    if errorlevel 1 ( echo [ERROR] see the log & set "OVERALL_RC=1" )
)

Three things in there are worth copying:

• Each item is a separately quoted argument. Joining item paths with semicolons does not work: GeoDmsRun then sees one long, unresolvable item path.
• The study area comes from the environment, not from an edited file. The batch sets STUDY_AREA per iteration and the configuration picks it up with parameter<string> StudyArea_ext := Expand(., '%%env:STUDY_AREA%%');, so the repository file is never touched by a run.
• /L<logfile> per step, and if errorlevel 1 after every call. GeoDmsRun returns a non-zero exit code on failure; without the check a later step happily consumes the output of a step that never produced one.

The solver is then invoked by the same driver, redirecting its output to a log:

julia --startup-file=no --threads=%THREADS% lambda_sweep_simplex.jl > "%LOG%" 2>&1

reading the solution back

The results come back as ordinary read storages. Because the batch guarantees that Julia has already finished, no ExitCode construction is needed here - the ordering is the batch file's job. Wrapping the read in a template lets one definition serve every solver variant:

Template ReadResults_T
{
   parameter<string> type;       // 'lp' or 'greedy'
   parameter<string> assignment; // 'central' or 'nearest'
   ///
   parameter<string> GDAL_DRIVER := 'Arrow';

   unit<uint32> ResultingClientAssignment
   :  StorageName     = "='%localDataProjDir%/'+ProjName+'/'+ModelParameters/StudyArea+'/'+type+'/'+assignment+'/traveltime.arrow'"
   ,  StorageType     = "gdal.vect"
   ,  StorageReadOnly = "True"
   {
      attribute<min_f> t_ij;
   }

   attribute<min_f> ClientTravelTime (Client) :=
      rjoin(ID(Client), convert(ResultingClientAssignment/id, Client), ResultingClientAssignment/t_ij);
}

container SolverResults
{
   container lp_central     := ReadResults_T('lp'    , 'central');
   container lp_nearest     := ReadResults_T('lp'    , 'nearest');
   container greedy_central := ReadResults_T('greedy', 'central');
   container greedy_nearest := ReadResults_T('greedy', 'nearest');
}

The rjoin is the point. Reading a file produced elsewhere creates a new domain unit that does not match the configured Client domain, even with the same number of rows. Joining on the exported id is robust against the solver reordering or dropping rows, which a positional := Client calculation rule is not. See parquet for the lighter alternatives when row order is guaranteed.

see also

• exec_ec - what orders what, and why the write needs its own GeoDmsRun call
• R - the same mechanism for an R model
• GeoDMS-through-Python
• parquet - the other typed exchange format