advertise new models

[robkooper]

When a new model comes online in docker it should advertise itself so it can be picked up by the framework. The idea is for the models to use a special channel in rabbitmq (models). This channel is a broadcast exchange. There is a special container that will listen for these messages and register them with BETY so they become available in the web interface. This container can also keep track of all the models and display statistics, such as number of jobs processed, average time per job, number of containers per model etc.

I would like to propose a file to be included in each model container that contains more information than we currently track in BETY, but can be made available throgh the model_collection_container. This is something done in clowder as well. An example of this file in json is below:

{
    "name": "ED2.2",
    "type": "ED2"
    "version": "24e6df6",
    "description": "The Ecosystem Demography Biosphere Model (ED2) is an integrated terrestrial biosphere model incorporating hydrology, land-surface biophysics, vegetation dynamics, and soil carbon and nitrogen biogeochemistry.",
    "author": "Paul Moorcroft <emailaddress>",
    "contributors": ["lots of people here"],
    "modeltype": "ED",
    "links": [
      {
        "type": "git",
        "description": "Source code to ED",
        "url": "https://github.com/EDmodel/ED2.git"
      },
      {
        "type": "issues",
        "description": "Issues related to the ED mode",
        "url": "https://github.com/EDmodel/ED2/issues"
      },
      {
        "type": "documentation",
        "description": "Documentation of the ED model",
        "url": "http://example.com"
      },
      {
        "type": "docker",
        "description": "Docker image of the ED model",
        "url": "pecan/models-ed"
      }
    ],
    "citations": ["Moorcroft, P. R., Hurtt, G. C. and Pacala, S. W. (2001), A METHOD FOR SCALING VEGETATION DYNAMICS: THE ECOSYSTEM DEMOGRAPHY MODEL (ED). Ecological Monographs, 71: 557-586. doi:10.1890/0012-9615(2001)071[0557:AMFSVD]2.0.CO;2"]
}

[ashiklom]

Looks good to me, at least as a first pass. JSON has the advantages that (1) it's RabbitMQ's default message-passing format, and (2) PostgreSQL has native support for both plain-text and binary JSON columns.

Given (2), what do you think about adding a docker_heartbeat JSON column to the BETY models table? We could configure one of the containers (or a new postgres-docker container?) to listen on this queue and just write this heartbeat directly into that column. The column could then be queried directly by the web interface and other services.

That's just one idea, and not necessarily a good one. Curious to hear other thoughts.

[infotroph]

What would the relationship be between this file and the existing register.MODELNAME.xml files? Are the use cases similar enough to make if confusing if both exist at once?

[robkooper]

@infotroph we can combine them. We just need to add the appropriate information to the json file proposed above.

@ashiklom we can can do store this in the model table. I was thinking of using that static document as a subdocument of a generated document:

{
   "id": "unique ID, can use $HOSTNAME in docker",
   "uptime": long_number_of_seconds_since_started,
   "jobs": long_number_of_jobs_executed,
   "avg_time": float_number_of_seconds_per_job
   "model_info": { document above }
}

Now we can have a list of all models, including how many there are running and some statistics. There might be more information we can add here as well. I think the information above is nice to capture, and we can maybe store in non BETY database for persistance (or just a file on disk), but this information is not required to run PEcAn.

[github-actions]

This issue is stale because it has been open 365 days with no activity.

[infotroph]

@robkooper is this still in progress? Not all models (yet?) have a model_info.json and we haven't (yet?) foldedregister.*.xml into the json, but the basic mechanism is all implemented, yeah?

[robkooper]

this is available for those models that run in docker. There is another issue #2567 of where this file should live.