Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parallel job rendering #91

Closed
hholzgra opened this issue Jul 3, 2022 · 2 comments
Closed

Parallel job rendering #91

hholzgra opened this issue Jul 3, 2022 · 2 comments
Labels
enhancement

Comments

@hholzgra
Copy link
Owner

hholzgra commented Jul 3, 2022

Right now rendering only really works in a single threaded way, even though the underlying render library can render multiple maps at the same time.

Different ideas to run this as a system service:

  • re-visit / re-enable the original multi threaded implementation
  • run multiple service instances that simply process requests by different criteria, each of them still being single threaded; e.g having separate services for single page, multi page, and API requests)
  • similar to the previous item, but have a "regular" and a "slow" queue, with jobs being moved to "slow" queue after some time
@hholzgra
Copy link
Owner Author

hholzgra commented Jul 3, 2022

Additional TODOs:

  • how to display multiple queue stats in map list, status overview ...

@hholzgra
Copy link
Owner Author

hholzgra commented Aug 16, 2022 •
edited
Loading

Current plan:

  • add "queue" field to rendering job model
  • display queue name in addition to position in list and detail view
  • modify to_render() method to accept optional queue name
  • add command line argument parsing to daemon.py to allow to specify queue name when calling
  • add queue name to queue position indicator in list view
  • file multi page layout requests to separate "multipage" queue
  • make render status check multi-queue aware
  • have new requests automatically filed into "default", "api" or "multipage" queue
  • show queue name in map details
  • allow to filter list by queue name
  • have default systemd service render everything regardless of queue name?

hholzgra added a commit that referenced this issue Aug 16, 2022
@hholzgra
add "queue" field to maprenderingjob (See Issue #91)
7bafbe8
hholzgra added a commit that referenced this issue Aug 16, 2022
@hholzgra
add support for multiple render queues (Issue #91)
e05711b 
* maprenderingjob model now has a "queue" name column,
  defaulting to "default"
* methods dealing with the rendering queue now all either
  use the jobs queue column value if already available
  or have an additional optional "queue_name" parameter
  with default value "default"
* a systemd template service file was added to be able to start
  individual per-queue render services
* the progress bar in the map request detail view was finally
  fixed to show actual position-in-queue progress
hholzgra added a commit that referenced this issue Aug 16, 2022
@hholzgra
show queue name together with queue position in list view (Issue #91)
218f4f5
hholzgra added a commit that referenced this issue Aug 17, 2022
@hholzgra
adjust system status displays to take multiple queues into account (I…
414ed5e 
…ssue #91)
@hholzgra hholzgra closed this as completed Sep 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@hholzgra