Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Real-time usage tracking #248

Closed
kylechard opened this Issue Apr 27, 2018 · 1 comment

Comments

Projects
None yet
3 participants
@kylechard
Copy link
Collaborator

kylechard commented Apr 27, 2018

For DESC we need to be able to specify a remote location to send usage information. The main item here is to develop a model for providing real-time information from Parsl to a DESC database (probably running in the cloud). We will work with the DESC team to make this data available for a UI to display the information to users.

From a configuration perspective the user should be able to select the DESC config (which defines the remote storage location).

Here is the schema we have come up with:

Workflow level

  • Workflow name (default: script name)
  • Execution start time, completion time
  • Executing user (linux username, in the future we could get Globus identity information)
  • Site name(s)
  • Site hostname(es)
  • Summary statistics (rolled up from the task level)
  • Number of tasks completed/failed/pending
  • Number of executions of this workflow
  • Version (if defined)
  • Link to runinfo folder (submit scripts, job info, etc.)

Task level

  • App name
  • State: completed/failed/pending + optional reason
  • Type: Bash/Python
  • Start time, completion time
  • Task ID
  • Input files (path)
  • Output files (path)
  • Link to logs (path)
  • Logs (referenced and accessible via Globus)
  • Stdout
  • Stderr
  • Runinfo
  • Controller log
  • Submit scripts
  • Job info
@yadudoc

This comment has been minimized.

Copy link
Member

yadudoc commented Jul 9, 2018

#371 fixes this.

@yadudoc yadudoc closed this Jul 11, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.