Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move batch creation to messaging system #1029

Closed
dfaller opened this issue Dec 15, 2017 · 2 comments
Closed

Move batch creation to messaging system #1029

dfaller opened this issue Dec 15, 2017 · 2 comments

Comments

@dfaller
Copy link
Contributor

dfaller commented Dec 15, 2017

Move the batch creation process of creating jobs and recipes to the messaging system for scalability.

Look at removing the usage (where possible) of the batch_job and batch_recipe models in favor of the job and recipe batch_id field. If possible, consider a plan to remove the batch_job and batch_recipe models.

@dfaller dfaller added this to the Backlog milestone Dec 15, 2017
@dfaller dfaller added this to To Do in Scalability via automation Dec 15, 2017
@dfaller dfaller moved this from To Do to In Progress in Scalability Feb 7, 2018
@dfaller dfaller modified the milestones: Backlog, Sprint 02-12-2018 Feb 7, 2018
@dfaller dfaller self-assigned this Feb 7, 2018
@dfaller
Copy link
Contributor Author

dfaller commented Feb 7, 2018

Updates to command messages:

  1. update_recipes - create job models that are ready, new recipes will create any jobs not already brought forward from superseded recipes, will now need to lock recipe models, punting on new recipe jobs pointing to superseded jobs because when doing recipe metrics will create one summary JSON field for recipe that holds metrics per job_name which will include superseded job IDs for use in creating new jobs
  2. unpublish_jobs - new command message to perform job product unpublishing and superseding, make a comment in publish_job so that when we have multiple jobs per job_name, we appropriately unpublish/supersede products from superseded recipe when no direct superseded job link occurs
  3. cancel_jobs - new command message to cancel jobs
  4. process_recipe_input - new command message to process 100 recipes' input files, will send update_recipes messages
  5. reprocess_recipes - new message that takes 100 recipes and supersedes them with new recipes, marks old recipe jobs as superseded, will need to create recipe_job models for jobs that are carried forward, sends process_recipe_input messages for new recipes and sends unpublish_jobs and cancel_jobs messages for superseded jobs

Update the batch creation logic to use the reprocess_recipes message.

The portion of the batch creation for new files will not be transitioned to use messaging. It will ultimately be replaced when batches are created from data sets.

@dfaller
Copy link
Contributor Author

dfaller commented Feb 7, 2018

Updates for batch_job and batch_recipe models:

  1. Stop creating batch_job and batch_recipe models.
  2. Transition logic using the models to use the batch_id field in job and recipe models.
  3. The batch_job and batch_recipe models will remain until we remove them from the database update.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
No open projects
Scalability
  
Done
Development

No branches or pull requests

1 participant