♻️ Scanner and background task rewrite (#265)

* 🎉 Start stateful job rewrite

scuffed

* 🚧 Restore job state

Lots of foundational work

* woof

* Fix incorrect schema items

* Update serde and tokio

* Add worker count config variable

* Add dunder directory for personal progress

* More progress

* Further JobManager progress

* Additional structuing on new manager.

* Additional logic for worker

* Update cargo lock

* Create thumbnail job example

* Add job manager to context and prune context struct.

* General cleanup

* nothing but scuff

* roughing out scanner with new system

* comment

* little bit by little bit

* Tweak scanner logic and add bench

* WIP use criterion for benches

* Benchmarks work!

* WIP subtasks

* WIP persist completed state and main run loop

* WIP use new impl

* WIP progress emitting

* WIP convert thumbnail job

* Push up last nights marathon

* Live log viewer

* Add migration

* WIP get UI job tracking functional again

* WIP cleanup and refactor

* WIP pause/resume

* WIP cleanup and refactor

* Refactor, reorg, clean

* Fix warning

* WIP: start using jms thumbnail manager

* Use the thumbnail manager during job

* WIP impl for missing series_job task

* WIP: consolidate

* WIP: consolidate

* Commit and push cleanup from last night, prep for experimental

---------

Co-authored-by: Joseph Micheli <josephwmicheli@gmail.com>

.github/actions/build-docker/action.yml

@@ -81,7 +81,7 @@ runs:
         tags: ${{ env.TAGS }}
 
     - name: Discord notification
-      if: ${{ success() && inputs.push == 'true' }}
+      if: ${{ success() && inputs.push == 'true' && inputs.discord-webhook != '' }}
       env:
         DISCORD_WEBHOOK: ${{ inputs.discord-webhook }}
       uses: 'Ilshidur/action-discord@0.3.2'

.github/workflows/experimental.yml

@@ -0,0 +1,34 @@
+name: 'Stump Experimental CI'
+
+on:
+  push:
+    branches:
+      - experimental
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+env:
+  DOCKER_USERNAME: ${{ secrets.DOCKERHUB_USERNAME }}
+  DOCKER_PASSWORD: ${{ secrets.DOCKERHUB_TOKEN }}
+
+jobs:
+  nightly-docker-build:
+    name: Build docker image
+    runs-on: [self-hosted]
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v3
+
+      - name: Setup and build docker image
+        uses: ./.github/actions/build-docker
+        with:
+          username: ${{ env.DOCKER_USERNAME }}
+          password: ${{ env.DOCKER_PASSWORD }}
+          tags: 'experimental'
+          load: false
+          push: true
+          platforms: 'linux/arm64/v8,linux/amd64'
+          # platforms: 'linux/arm64/v8,linux/arm/v7,linux/amd64'
+          discord-webhook: ''

.gitignore

@@ -3,6 +3,7 @@
 
 # Logs
 logs/
+!packages/browser/**/logs/
 *.log
 
 # Cache

.prettierignore

@@ -10,4 +10,4 @@ build
 .next
 
 # locale files
-interface/src/i18n/locales/*.json
+packages/browser/src/i18n/locales/*.json