Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add engine benchmark analysis tool #5852

Merged
merged 15 commits into from
Mar 28, 2023
Merged

Conversation

Akirathan
Copy link
Member

@Akirathan Akirathan commented Mar 8, 2023

Pull Request Description

Add Engine benchmark analysis tool - a python script for downloading benchmark data, and Enso project for the analysis. I have also included benchmark data for 02/2022.

Related issues and discussions:

Important Notes

Checklist

Please include the following checklist in your PR:

  • The documentation has been updated if necessary.
  • All code conforms to the
    Scala,
    Java,
    and
    Rust
    style guides.
  • All code has been tested:
    • Unit tests have been written where possible.
    • If GUI codebase was changed: Enso GUI was tested when built using BOTH
      ./run ide build and ./run ide watch.

@Akirathan Akirathan requested a review from hubertp March 8, 2023 11:57
@Akirathan Akirathan added the CI: No changelog needed Do not require a changelog entry for this PR. label Mar 8, 2023
@Akirathan Akirathan self-assigned this Mar 8, 2023
Copy link
Member

@JaroslavTulach JaroslavTulach left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for sharing your work, @Akirathan. I believe it is a great start. However right now the Enso project doesn't work for me due to "expecting Text, but got ZonedDateTime":

Broken

Tested on clean build of IDE, engine and project-manager made from commit 5c19c00

@JaroslavTulach
Copy link
Member

@jdunkerley, @Frizi - how shall we achieve some "level of interactiveness" in Enso IDE?

My idea was to create a table with columns representing each benchmark:

empty table with columns

the table is empty, but contains all the benchmark names as columns. Now I'd like to let the Enso user interactively select one of the columns - is that possible? I hoped it is, but I see no "drop down widget" to select from the column names.

The missing functionality may be related to #5631 - but I don't need multi choice that much. Any other idea how to let user select from a list of benchmarks? Shall I create a function and annotate it inside of Main.enso with some @widget annotation?

@radeusgd
Copy link
Member

radeusgd commented Mar 9, 2023

@jdunkerley, @Frizi - how shall we achieve some "level of interactiveness" in Enso IDE?

My idea was to create a table with columns representing each benchmark:

empty table with columns

the table is empty, but contains all the benchmark names as columns. Now I'd like to let the Enso user interactively select one of the columns - is that possible? I hoped it is, but I see no "drop down widget" to select from the column names.

The missing functionality may be related to #5631 - but I don't need multi choice that much. Any other idea how to let user select from a list of benchmarks? Shall I create a function and annotate it inside of Main.enso with some @widget annotation?

If you just need a single column, maybe Table.at would work?

@JaroslavTulach
Copy link
Member

@jdunkerley, @Frizi - how shall we achieve some "level of interactiveness" in Enso IDE?
If you just need a single column, maybe Table.at would work?

I don't see any dropdown:

No drop down for Table.at

Which is strange, as there is @selector on Table.at.

@Frizi
Copy link
Contributor

Frizi commented Mar 10, 2023

@JaroslavTulach It looks like a regression on engine side or libs. On latest nightly (2023.1.1-nightly.2023.3.9), the widget backing visualization returns an error, so the widget doesn't show up. The error is reported in the console, though you have to call showLogs() to see it.

Reported visualization error: Method `map` of Function could not be found.
{
  "jsonrpc": "2.0",
  "method": "executionContext/visualisationEvaluationFailed",
  "params": {
    "contextId": "6d8fccce-38b3-495a-87ed-4af06a67ddb5",
    "visualisationId": "41fec3bf-eebc-44d7-a185-bca5753a33f0",
    "expressionId": "c85c515e-cc9e-4022-a826-f4faeb6db66d",
    "message": "Method `map` of Function could not be found.",
    "diagnostic": {
      "kind": "Error",
      "message": "Method `map` of Function could not be found.",
      "path": {
        "rootId": "19e3e980-e2f7-4b4c-bf20-5753a2e4aaf7",
        "segments": [
          "src",
          "Main.enso"
        ]
      },
      "location": {
        "start": {
          "line": 17,
          "character": 16
        },
        "end": {
          "line": 17,
          "character": 56
        }
      },
      "expressionId": "8148e959-14dc-4530-b0d8-58e6b832b0e2",
      "stack": [
        {
          "functionName": "Main.main",
          "path": {
            "rootId": "19e3e980-e2f7-4b4c-bf20-5753a2e4aaf7",
            "segments": [
              "src",
              "Main.enso"
            ]
          },
          "location": {
            "start": {
              "line": 17,
              "character": 16
            },
            "end": {
              "line": 17,
              "character": 56
            }
          },
          "expressionId": "8148e959-14dc-4530-b0d8-58e6b832b0e2"
        }
      ]
    }
  }
}

On older engine version (2023.1.1-nightly.2023.3.4) it works.
image

@sylwiabr
Copy link
Member

@Frizi can you create an issue for this regression?

@Akirathan
Copy link
Member Author

bench_download.py should now be more or less ready to use. You can try to use it and report any errors you encounter. In the help bench_download.py -h it should be explained what to do.

@jdunkerley jdunkerley linked an issue Mar 14, 2023 that may be closed by this pull request
6 tasks
Copy link
Member

@JaroslavTulach JaroslavTulach left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Getting started experience isn't easy due to missing documentation (needed to run the script with --help to get some documentation).

@Akirathan
Copy link
Member Author

Updated the PR - Added some documentation, and general fixes like creating parent directories. I will try to merge this ASAP and create some follow-up issues for comparison of multiple branches.

@Akirathan Akirathan added the CI: Ready to merge This PR is eligible for automatic merge label Mar 28, 2023
@mergify mergify bot merged commit 3ca0a17 into develop Mar 28, 2023
@mergify mergify bot deleted the wip/akirathan/engine-benchs branch March 28, 2023 15:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI: No changelog needed Do not require a changelog entry for this PR. CI: Ready to merge This PR is eligible for automatic merge
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Comparing Enso benchmarks results with Enso
6 participants