Several tools will require an archive that provides access to simulations provided by rubin_scheduler. For example, the prenight briefing dashboard supplied by schedview is a tool for visualizing simualtions of a night of observing, and it requires access to such simulations. There will eventually be other dashboards that will also need to read such tables for visualization, as well as tables describing visits completed by the actual instruments. Users of these dashboards will need to use them to select which data sets are to be visualised: the dashboard code needs to provide both user interface elements that let the user select the desired table of visits, and also actually load the table itself. The dashboard servers run within containers on kubernetes-based infrastructure, which operate best when persistent data is stored outside the containers themselves. Therefore, these dashboards require an external (to the container) resource that supports searching for available simulations, and access to the data itself.
This archive design is intended primarily as a prototype, something to experiment with for a better informed development of requiriments. So, flexibility and speed of implementation have been prioritized, with the intention that there be a significant rofactoring (or even outright replacement) when requirements have been more thoroughly developed.
The archive itself is a directory tree. The URI of an archive is the URI of the root of the directory tree as supported by the lsst.resources package. Each simulation in the archive has its own directory in this directory tree, named according to the ISO-8601 date on which a simulation was added to the archive, and a simple incrementing index separating different simulations added on the same day, such that the format for the directory for a specific simulation is:
${ARCHIVE_URI}/{ISO_DATE}/{ADDITION_INDEX}For example, if the URI of the archive is:
file:///my_data/sim_archivethen the URI of the third simulation added on June 21, 2030 will be:
file:///my_data/sim_archive/2030-06-21/3Each simulation directory contains a metadata yaml file named sim_metadata.yaml. In the above example, the URI for this metadata file would be:
file:///my_data/sim_archive/2030-06-21/3/sim_metadata.yamlA minimal sim_metadata.yaml file specifies the name of the sqlite3 database file with the visits. For example, if the URI for the visit database in the above example is file:///my_data/sim_archive/2030-06-21/3/opsim.db, then the minimal content of sim_metadata.yaml would be:
files:
observations:
name: 'opsim.db'All other data in the metadata file is optional, but additional metadata will be required if the archived simulation is to be used for some use cases. For example, if schedview's prenight dashboard is to be able to load the reward data, it must be able to locate the reward data from the metadata file, so that the metadata file needs to look something like this:
files:
observations:
name: 'opsim.db'
rewards:
name: 'rewards.h5'Clients of the archive will also need to search available simulations for those meeting relevant criteria. For example, the prenight dashboard will seach for simulations the include a desired night, in which case the range of nights covered by the simulation must be included.
A sample metadata file that includes an early guess at what the prenight dashboard will use looks like this:
files:
observations:
name: opsim.db
rewards:
name: rewards.h5
label: Notebook test on 2024-01-04 16:49:44.299
simulated_dates:
first: '2025-05-05'
last: '2025-05-05'In the above:
labelSimulations will appear in drop-down section widgets in dashdoards such as the pre-night dashboard. The
labelelement in the determines how the simulation will appear in the dropdown. In other applications, this element may also be used as plot annotations or column or row headings.simulation_datesShows the range of dates covered by the simulation. When the user specifies a night, the
prenightdashboard will restrict the offered to those that cover the specified date.
Finally, a number of other elements may be included for debugging purposes. A full file might look something like this:
files:
environment:
md5: 4381d7cc82049141c70216121e39f56d
name: environment.txt
notebook:
md5: 6b75c1dd8c4a3b83797c873f3270cc04
name: notebook.ipynb
observations:
md5: 1909d1afaf744ee50bdcf1a9625826ab
name: opsim.db
pypi:
md5: 9c86ea9b4e7aa40d3e206fad1a59ea31
name: pypi.json
rewards:
md5: 6d3c9d3e0dd7764ed60312e459586e1b
name: rewards.h5
scheduler:
md5: 5e88dfee657e6283dbc7a343f048db92
name: scheduler.pickle.xz
statistics:
md5: c515ba27d83bdbfa9e65cdefff2d9d75
name: obs_stats.txt
label: Notebook test on 2024-01-04 16:49:44.299
simulated_dates:
first: '2025-05-05'
last: '2025-05-05'
scheduler_version: 1.0.1.dev25+gba1ca4d.d20240102
sim_runner_kwargs:
mjd_start: 60800.9565967191
record_rewards: true
survey_length: 0.5155218997970223
tags:
- notebook
- devel
host: neilsen-nb
username: neilsenThis example has a number of additional elements useful for debugging, and which pehaps might be useful for future applictions, but which are not used (or planned to be used) by the prenight dashboard.
files/*A number of other types of files associated with specific simulations may be included. These may be useful in future applications, or for debugging only. See below for descriptions of the extra types of files in this example.
files/${TYPE}/md5Checksums for various files. These can be useful both for checking for corruption, and for determining whether two simulations are identical without needing to download either.
scheduler_versionThe version of the scheduler used to produce the simualtions.
sim_runner_kwargsThe arguments to the execution of
sim_runnerused to run the simulation.tagsA list of ad-hoc keywords. For example, simulations used to test a specific jira issue may all have the name of the issue as a keyword. Simulations used to support a give tech note may have the name of the tech note.
hostThe hostname on which the simulation was run.
usernameThe username of the user who ran the simulation.
Optional (for debugging or speculative future uses only) file types listed above are:
environmentThe conda environment specification for the environment used to run the simulation.
notebookThe notebook used to create the simulation, for example as created using the
%notebookjupyter magic.pypyThe
pypypackage list of the environment used to run the simulation. If the simulation is run using only conda-installed packages, this will be redundant withenvironment.schedulerA python pickle of the scheduler, in the state as of the start of the simulation.
statisticsBasic statistics for the visit database.
The rubin_scheduler package provides a tool to combine running a simulation and adding the results to an archive, including any metadata that can be derived automatically. The rubin_scheduler.sim_archive.drive_sim function is wrapper around rubin_scheduler.scheduler.sim_runner that incorporates this metadata collection and the creation of the entry in an archive. It takes all of the same arguments that sim_runner does, and passes them directly to sim_runner. In addition, it takes a few arguments that specify the archive into which it is to be added (archive_uri), the label to be included in the metadata (label), and the code used to run the simulation (ethier script or notebook). Details are available in the drive_sim docstring.
For example, if this code is put into a file and run as a script, it will run the specificed simulation and add it to the specified archive:
from astropy.time import Time
from rubin_scheduler.scheduler.example import example_scheduler
from rubin_scheduler.scheduler.model_observatory import ModelObservatory
from rubin_scheduler.sim_archive import drive_sim
sim_mjd_start = Time("2025-05-05").mjd + 0.5
# The start date of the simualtion.
# Offset by 0.5 to avoid starting late when the MJD rollover occurs during or
# after twilight. See dayObs in SITCOMTN-32: https://sitcomtn-032.lsst.io/ .
sim_length = 1.0
# Passed to sum_runner, in units of days.
archive_uri = "file:///sdf/data/rubin/user/neilsen/data/test_sim_archive/"
# The URI of the root of the archive. The trailing "/" is required.
observatory = ModelObservatory()
scheduler = example_scheduler()
scheduler.keep_rewards = True
results = drive_sim(
observatory=observatory,
scheduler=scheduler,
archive_uri=archive_uri,
label=f"Example simulation started at {Time.now().iso}.",
script=__file__,
tags=["example"],
mjd_start=sim_mjd_start,
survey_length=sim_length,
record_rewards=True,
)The result looks like this:
bash$ ls /sdf/data/rubin/user/neilsen/data/test_sim_archive/2024-01-18/1
environment.txt example_archived_sim_driver.py obs_stats.txt opsim.db pypi.json rewards.h5 scheduler.pickle.xz sim_metadata.yaml
bash$ cat /sdf/data/rubin/user/neilsen/data/test_sim_archive/2024-01-18/1/sim_metadata.yaml
files:
environment:
md5: 33f94ddf8975f9641a1f524fd22e362e
name: environment.txt
observations:
md5: 8b1ee9a604a88d2708d2bfd924ac3cd9
name: opsim.db
pypi:
md5: 51a8deee5018f59f20d5741fd1a64778
name: pypi.json
rewards:
md5: 10e4ab9397382bfa108fa21354da3526
name: rewards.h5
scheduler:
md5: 35713860dc9ba7a425500f63939d0e02
name: scheduler.pickle.xz
script:
md5: b4a476a4fd1231ea1ca44149784f1c3f
name: example_archived_sim_driver.py
statistics:
md5: 7c6a6af38aff3ce4145146e35f929b47
name: obs_stats.txt
host: sdfrome002.sdf.slac.stanford.edu
label: Example simulation started at 2024-01-18 15:46:27.758.
scheduler_version: 1.0.1.dev25+gba1ca4d.d20240102
sim_runner_kwargs:
mjd_start: 60800.5
record_rewards: true
survey_length: 1.0
simulated_dates:
first: '2025-05-04'
last: '2025-05-04'
tags:
- example
username: neilsenAlternately, the simulation can be run from a jupyter notebook similarly, excepet that instead of saving the script that generated the simulation, a notebook with the cells of notebook that created the simulation up to the cell that runs the simulation can be stored instead. An example can be found in the archive/sim_and_archive.ipynb in the rubin_sim_notebook github repository.