Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Run a recipe over multiple inputs #566

Merged
merged 180 commits into from Jul 4, 2019

Conversation

Projects
None yet
2 participants
@j433866
Copy link
Member

commented May 31, 2019

This PR adds the ability to load multiple inputs into the web app (#505) , and run a recipe over all of them. I have also made some major changes to the input/output/worker waiters to work with multiple inputs, and have moved a lot of the functionality from the input waiter into a new worker called InputWorker. There are also some improvements to the UI (new themes!)

Multiple inputs

  • The input/output area now has tabs! You can create a new tab in the input, either by loading in multiple files or adding a tab manually. Each tab can be either a text input or loaded file.
  • When loading multiple files, a pool of LoaderWorkers is created to load them as quickly as possible, and when bake is clicked, a pool of ChefWorkers is used to utilise multi-processor systems. This uses the navigator.hardwareConcurrency value. (Or 4 if this is unavailable)
  • You can also now download all the outputs in a zip file, which uses a new worker (ZipWorker) to zip up the files in the background before downloading them. When zipping, we can use Magic to automatically detect the file extension for each file. (Defaults to .dat when we can't detect it)
  • There is a new dialog for searching for input and output tabs, where you can filter by status (pending / loading / loaded for inputs, and pending / baking / baked / stale / errored for outputs), and also regex the contents of the tab
  • The majority of the logic for the inputWaiter has been moved into a new worker (InputWorker), as the browser would frequently hang when handling a large amount of inputs. Now, it should handle a large number reasonably smoothly. Generally we can't load more than around 500,000 at once, the browsers tend to get quite unhappy after that!
  • Also, the bake button is now a progress bar!
    bake

Other UI Improvements

  • Added an input debouncer to avoid baking too often
  • Images loaded into the input can show a preview instead of the file icon (this can be disabled)
  • Added a new open folder button for opening an entire folder (and its subfolders)
  • Added new solarized light and dark themes

Solarized light and dark themes

j433866 added some commits Mar 19, 2019

Add new solarized light and dark themes.
Add more elements to be controlled by theme css:
- Preloader spinner colours
- Operation disable / breakpoint icons
- Auto bake checkbox
- Search highlight colour
- Categories header colour
Add styling of checkboxes.
Change fonts in solarized themes to match classic.
Add styling of input fields to modals
Add UI for input tabs.
Can add, remove and switch tabs
Stores inputs in memory instead of in textareas.
Allow selecting multiple files.
Allow drag and dropping multiple files.
Clear all IO button will close all tabs.
Will now spawn multiple loaderworkers simultaneously.
Improve tab logic to break less and work in Firefox.
Show tab number on tab title.
Don't show lines value if tab contents is a file.
Bake all inputs, not just the active tab.
Limits number of workers to number of cpu threads (4 if not supported)
Creates output tabs (switching doesn't work yet)
Disabled some highlighting for now.
Sync output tabs to input tabs.
Output tabs are created when input tabs are.
Fix initial bake not working
Rewrite InputWaiter to be less messy.
Don't create a DOM element for every tab, just reuse the same ones.
Display file information while the files are loading.
(Output tabs no longer work)
Change "input-tabs" id to be for the actual ul element.
Automatically disable auto-bake if more than 20 files are input.
Add tab buttons (don't work yet).
Change some of the HTML around the tab bar
Improve WorkerWaiter and OutputWaiter.
- Will run and display outputs in the output area as they're baked
- Creates output tabs
- Can change output tabs (only the first 4 at the moment)
Output tabs now (mostly) work.
Add downloading files as an archive (needs work)
Add option for keeping the tabs in sync
Fix some weirdness with tabs.
Adjust worker logic to avoid closing and reopening a worker as often
Fix more tab weirdness.
Move tab buttons onto tab bar.
Calculate size of maxTabs automatically on page load.
Display total execution time when a bake finishes.

j433866 added some commits Jun 13, 2019

Add mime type detection to tab title.
Move addition of close tab button back
Improve queueing for getDish... messages.
Force wheel event to be not passive.
Change maxworkers to be navigator.hardwareConcurrency - 1
Fix switchClick not always being fired.
Run detectFileType on switched output

@n1474335 n1474335 changed the base branch from master to v9 Jun 19, 2019

@n1474335
Copy link
Member

left a comment

Awesome stuff as always. A few things to review here, then it'll be ready to merge.

Show resolved Hide resolved src/core/ChefWorker.js Outdated
Show resolved Hide resolved src/core/ChefWorker.js Outdated
Show resolved Hide resolved src/core/Dish.mjs Outdated
Show resolved Hide resolved src/core/Dish.mjs Outdated
Show resolved Hide resolved src/core/Dish.mjs Outdated
Show resolved Hide resolved src/web/workers/InputWorker.mjs
Show resolved Hide resolved src/web/workers/InputWorker.mjs
Show resolved Hide resolved src/web/workers/InputWorker.mjs
Show resolved Hide resolved src/web/workers/InputWorker.mjs
Show resolved Hide resolved src/web/workers/InputWorker.mjs Outdated

@n1474335 n1474335 merged commit 6964bbc into gchq:v9 Jul 4, 2019

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
license/cla Contributor License Agreement is signed.
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.