Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[22.05] Various fixes for running pulsar jobs #15919

Merged
merged 6 commits into from
Apr 10, 2023
Merged

[22.05] Various fixes for running pulsar jobs #15919

merged 6 commits into from
Apr 10, 2023

Conversation

mvdbeek
Copy link
Member

@mvdbeek mvdbeek commented Apr 10, 2023 •
edited

Fixes:

  • extra_file handling when remote metadata is turned off in pulsar
  • extra_file handling when pulsar stages files back to object store
  • determining job messages when remote metadata is turned off in pulsar
  • automatically turns off parallelism when runner is pulsar

This should pass tests on its own, and together with galaxyproject/pulsar#318 all of the framework tests should pass.

How to test the changes?

(Select all options that apply)

  • I've included appropriate automated tests.
  • This is a refactoring of components with existing test coverage.
  • Instructions for manual testing are as follows:
    1. [add testing steps and prerequisites here if you didn't write automated tests covering all your changes]

License

  • I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

@github-actions github-actions bot added this to the 23.0 milestone Apr 10, 2023
@mvdbeek
Fix extra files path if metadata job runs in celery
33f9edf 
The job runner may have moved files to the object store already before
calling `_handle_metadata_externally`, so the
celery task needs to process files from the object store, not the job
working directory.
@mvdbeek
Test tools for extra file handling in pulsar embedded test case
8019207
@mvdbeek
Write exit code file when when setting metadata outside of pulsar
e29f927 
We need this to figure out the right stdout/stderr and job state with
the `check_output` function.
@mvdbeek
Automatically disable parallelism on pulsar job runner
ef6812d
@mvdbeek mvdbeek merged commit 387660b into galaxyproject:release_22.05 Apr 10, 2023
29 of 36 checks passed
@github-actions
Copy link

This PR was merged without a "kind/" label, please correct.

@mvdbeek mvdbeek mentioned this pull request Apr 10, 2023
Closed
4 tasks
@nsoranzo nsoranzo deleted the pulsar_fixes_22_05 branch April 10, 2023 22:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dannon dannon dannon approved these changes

@jmchilton jmchilton jmchilton approved these changes

Assignees
No one assigned
Labels
area/jobs area/testing/integration area/testing kind/bug kind/enhancement
Projects
None yet
Milestone
23.0
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@mvdbeek @dannon @jmchilton