Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Requeueing #307

Merged
merged 63 commits into from
Aug 9, 2023
Merged

Requeueing #307

merged 63 commits into from
Aug 9, 2023

Conversation

JasonWildMe
Copy link
Contributor

Bringing a better pipeline to Wildbook-WBIA interaction

holmbergius and others added 30 commits July 12, 2023 20:34
@holmbergius
First pass separate detection queue
a8d70b3
@holmbergius
Help a bulk import self-heal missing acmIDs
96fe902
@holmbergius
Make the asset check much faster
a830591
@holmbergius
add missing shepherd close
5989b69
@holmbergius
Reduce time wait
d1a2411
@holmbergius
detection jobs flowing around ID jobs in queue
cd64a78
@holmbergius
Remove need for expensive AssetStore checks
560b8ac
@holmbergius
Help a bulk import self-heal missing acmIDs
4a688d3
@holmbergius
Pretty awesome spee improvement
11d1b04
@holmbergius
Part 1: open data integrity checks to researchers
6f013a3
@holmbergius
Part 2: Check display of values and admin override from param only
eb8c446
@holmbergius
Restore unexpected /ia target from iaResults.jsp
eb1b49f
@holmbergius
Add admin override
3b8617b
@naknomum
toggle jobs queue based on ident vs detect
ea55693
@holmbergius
Provide more informative details for dupes
6f9a737
@naknomum
requeue
d2110f3
@naknomum
allow for no incrementing (by default); also have a timeout (currentl…
cb91872 
…y 2 days)
@naknomum
forgot one
e667d82
@holmbergius
More conservative Shepherd handling
68acac1
@holmbergius
Add connection checking to avoid a far future timeout from Rabbit<Q
61e9969
@holmbergius
Better clumping same species detection requests
15138eb
@holmbergius
Allow small batch jobs to use the fastlane queue
957c8e9
@holmbergius
Fastlane understood when tasks return as well
4d7971e
@holmbergius
Set up automated acmId fixing should any of these have experienced a …
c9f14fc 
…timeout
@holmbergius
Tweak exception message
4aebbaf
@holmbergius
Delete copied but unused imports
5c31211
@holmbergius
Provide summary stats and refine queries
57b44f2
@holmbergius
Merge branch 'self-healing-import-acmid' into requeueing
46c3c1b 
# Conflicts:
#	src/main/java/org/ecocean/media/AssetStore.java
#	src/main/webapp/import.jsp
@holmbergius
tweak tracking on processing
52c29ea
@holmbergius
Remove empty set location match (confusing results and slows query)
5998b29
holmbergius and others added 28 commits August 2, 2023 14:00
@holmbergius
table view is easier to read
cf4cac4
@naknomum
maybe actually bubble up to requeue now
2050a9e
@naknomum
background requeueing
cfd8377
@naknomum
more cautious
3c995c8
@holmbergius
Bulk import single detection task
d005f9a
@holmbergius
requeue fastlane too
17de42a
@holmbergius
Handle continuously failing/corrupt/weird images
3cd3868
@holmbergius
FIX: \\*
553b7f1
@holmbergius
Hushes an exception polluting the logs
8471f4a
@naknomum
catch failure for requeueing
c661335
@holmbergius
Don't requeue empty lists
5f7b86c
@holmbergius
Catch and loop weird situation where queue itself is unavailable
df8d64b
@holmbergius
Expose stats of matching progress
cc6c935
@holmbergius
Language tweaks
4d39f2b
@holmbergius
Remove unneeded println
78bb124
@holmbergius
Hush unneeded line
148bee2
@holmbergius
Report correct status where children have children
c512a0e
@holmbergius
Another JSON format
387c8f2
@holmbergius
More edge case status detection
876b6b8
@holmbergius
Make common code and better task future foundation
e7420b6
@holmbergius
Add queue size
dd7266f
@holmbergius
Remove unneeded line
0414ffd
@holmbergius
Add new Task parameter persistence
bdf2212
@holmbergius
Fix ORM mapping for Task
2cdc415
@holmbergius
Better track task loading and completion
f1a4c7f
@holmbergius
add fastlane ID metrics
9474516
@holmbergius
Merge branch 'open-data-integrity' into requeueing
fd7f27f
@holmbergius
Add a user visible queue snapshot
ea328d4
@JasonWildMe JasonWildMe merged commit 3d0ebf7 into master Aug 9, 2023
2 checks passed
@TanyaStere42 TanyaStere42 deleted the requeueing branch February 5, 2024 22:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@JasonWildMe @holmbergius @naknomum