Don't write job and job-related attributes when creating export model

[mvdbeek]

These aren't used in regular jobs (should only be needed for history export currently) and looping over the implicit collection jobs for each job is quadratic in runtime and inefficient.

What did you do?

• Skip serializing job and implicit collection job attributes when they are not needed

Why did you make this change?

@bgruening mentioned that when a user submits 10000 map-over jobs he starts seeing slow query timing logs.
When this happens all of the active job worker threads are within galaxy/model/__init__.py:1536

Thread 2747314 (idle): "CondorRunner.work_thread-3"
    do_execute (sqlalchemy/engine/default.py:593)
    _execute_context (sqlalchemy/engine/base.py:1277)
    _execute_clauseelement (sqlalchemy/engine/base.py:1130)
    _execute_on_connection (sqlalchemy/sql/elements.py:298)
    execute (sqlalchemy/engine/base.py:1011)
    _execute_and_instances (sqlalchemy/orm/query.py:3560)
    __iter__ (sqlalchemy/ext/baked.py:444)
    _load_on_pk_identity (sqlalchemy/ext/baked.py:615)
    _emit_lazyload (sqlalchemy/orm/strategies.py:850)
    <lambda> (<string>:1)
    _load_for_state (sqlalchemy/orm/strategies.py:760)
    get (sqlalchemy/orm/attributes.py:723)
    __get__ (sqlalchemy/orm/attributes.py:287)
    <listcomp> (galaxy/model/__init__.py:1536)
    serialize (galaxy/model/__init__.py:1536)
    _finalize (galaxy/model/store/__init__.py:1380)
    __exit__ (galaxy/model/store/__init__.py:1397)
    setup_external_metadata (galaxy/metadata/__init__.py:167)
    setup_external_metadata (galaxy/jobs/__init__.py:2106)
    __handle_metadata (galaxy/jobs/command_factory.py:235)
    build_command (galaxy/jobs/command_factory.py:128)
    build_command_line (galaxy/jobs/runners/__init__.py:285)
    prepare_job (galaxy/jobs/runners/__init__.py:244)
    queue_job (galaxy/jobs/runners/condor.py:65)
    run_next (galaxy/jobs/runners/__init__.py:136)
    run (threading.py:864)
    _bootstrap_inner (threading.py:916)
    _bootstrap (threading.py:884)

At this point Galaxy serializes all jobs that belong to an implicit collections jobs association.
Note that we only run through the export model by default since 21.01 (it was made the default in 7d0ec29).

How to test the changes?

(select the most appropriate option; if the latter, provide steps for testing below)

• I've included appropriate automated tests.
• This is a refactoring of components with existing test coverage.
• Instructions for manual testing are as follows:
  1. [add testing steps and prerequisites here if you didn't write automated tests covering all your changes]

License

• I agree to license these contributions under Galaxy's current license.
• I agree to allow the Galaxy committers to license these and all my past contributions to the core galaxy codebase under the MIT license. If this condition is an issue, uncheck and just let us know why with an e-mail to galaxy-committers@lists.galaxyproject.org.

For UI Components

• I've included a screenshot of the changes

[jmchilton]

That if looks iffy right? Maybe there should a TODO here and options used more clearly? I tried to make this code pretty general and it seems to be becoming more metadata specific slowly instead moving the other direction. for_edit doesn't really naturally seem to imply serialize job data. I feel like it might be better to add options for including job data and set at a different level.

[mvdbeek]

Do you think e7e3b00 looks better ?

[github-actions]

This PR was merged without a "kind/" label, please correct.

[bgruening]

Thanks a lot @mvdbeek!

[bgruening]

Ah shit ... you wanted to target 21.01?

[mvdbeek]

Ah, my bad ... yeah, I'll open a backport.

[bgruening]

Very nice, this works pretty well on EU! Thanks a lot @mvdbeek!