Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[EE] Discuss how libres and ERT should communicate #1061

Closed
sondreso opened this issue Oct 8, 2020 · 2 comments
Closed

[EE] Discuss how libres and ERT should communicate #1061

sondreso opened this issue Oct 8, 2020 · 2 comments
Assignees

Comments

@sondreso
Copy link
Collaborator

sondreso commented Oct 8, 2020

We need to find a suitable protocol for ERT and libres to communicate over. This should preferably not be something we invent ourselves. We should also make sure we solve this without creating a circular dependency between ERT and libres!

@jondequinor
Copy link
Contributor

jondequinor commented Oct 9, 2020

Doing a quick survey, it seems that https://cloudevents.io/ is a potential winner. They have also done a survey of existing event formats, for reference.

It's protocol agnostic, the events are JSON and have the following required attributes:

  • id
  • source
  • specversion
  • type

Example:

{
    "specversion" : "1.x-wip",
    "type" : "com.github.pull_request.opened",
    "source" : "https://github.com/cloudevents/spec/pull",
    "subject" : "123",
    "id" : "A234-1234-1234",
    "time" : "2018-04-05T17:31:00Z",
    "comexampleextension1" : "value",
    "comexampleothervalue" : 5,
    "datacontenttype" : "text/xml",
    "data" : "<much wow=\"xml\"/>"
}

There's a Python SDK https://github.com/cloudevents/sdk-python

Is well on its way of being blessed by the CNFC…

They provide a vocabulary with which we can talk about events https://github.com/cloudevents/spec/blob/master/spec.md#notations-and-terminology

@sondreso
Copy link
Collaborator Author

sondreso commented Oct 9, 2020

https://github.com/cloudevents/spec/blob/master/spec.md

All events have:
  time
  data
  source
  type
  id
  (subject)

Following events are defined

Forward Model Job

  - Start
    - data:
    - id
    - type: "ert.forward_model_job.start"
  - Running
    - data: max_memory_usage, current_memory_usage, std_err, std_out
  - Success
    - data:
  - Failure
    - data: exit_code (optional), error_msg
  - Event attributes:
    - id: wonder if this should just be UUID for the time being
    - source: /ert/job/<ee_id>/<real_id>/<step_id>/<job_id>
    - subject: 


{
  time: "2015-05-12…",
  data: "{max_memory_usage: 10000}",
  source: "/ert/job/0/23/",
  type: "com.equinor.ert.ensemble_evaluator.forward_model.job.start",
  id: "abcdef-123456-…-…",
}

Forward Model Step (Job dispatch)

(Local queue for executing job_dispatch, aka meta job_dispatch)

  - Init
    - data: jobs
  - Failure
    - data: error_msg
  - Success
    - data:
  - Event attributes:
    - id: UUID
    - source: /ert/forward_model/<run_id>:<real>:<step_id>
    - source: /ert/ee/<ee_id>/real/<real_id>/step/<step_id>

{
  time: "2015-05-12…",
  data: "{max_memory_usage: 10000}",
  source: "/ert/ee/<ee_id>/real/<real_id>/step/<step_id>",
  type: "com.equinor.ert.ensemble_evaluator.forward_model.step.init",
  id: "abcdef-123456-…-…",
}

Forward Model Stage (Queue Manager)

  - Start
  - Waiting
  - Pending
  - Running
  - Success
  - Failure
  - Unknown
  - Event attributes:
    - id: UUID
    - source: /ert/forward_model/<something_evaluator_unique>/real/<real_id>


{
  time: "2015-05-12…",
  data: "{}",
  source: "/ert/ee/<ee_id>/real/<real_id>",
  type: "com.equinor.ert.ensemble_evaluator.forward_model.stage.waiting",
  id: "abcdef-123456-…-…",
}

Ensemble Evaluator (TBD)

  - EnsembleResponseEvent
    - data: map(…)
  - Event attributes:
    - id: UUID
    - source: /ert/ensemble_evaluator/<uuid>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants