Create basic bulk import UI

[paynejd]

If possible, we would like to have a basic bulk import UI available before we publicly launch OCL v2. For a first pass, we'd like the following:

• Ability to view list of bulk imports in the queue (pending, in progress, recently completed, etc) - ideally this pulls the list across all supported import methods
• Simple interface to upload a bulk import file or paste in JSON, optionally set a custom queue, select any options (test mode, parallel, etc.) and submit the bulk import for processing
• Ability to track progress of bulk import(s) in the queue
• Ability to view/download the results of a bulk import

Future requirements:

• Validate bulk import JSON/file before submitting it
• Manage queue (cancel, pause, delete, re-submit, etc.)

@snyaggarwal @jamlung-ri @bmamlin @harpatel1 @rkorytkowski @maurya Anything to add here?

[jamlung-ri]

@paynejd I saw this on the API Plan document. Is it (or should it be) reflected in the list above?

Bulk Imports ability to import “diff files” that support all features required for reference vocabulary maintenance

Also, I am wondering where this UI can be accessed. I assume it would be at the global level (e.g. you can get to it from the OCL global search page or from the Profile menu). Maybe in the future, you can get to it from an existing Organization/User page or a Source/Collection page?

[snyaggarwal]

Tasks:

• Import List
  • List All bulk imports (parallel/non-parallel) order by received desc
  • Track status of running import (polling)
  • Color/Icon differentiator for each status (pending/started/revoked/failure/success
  • Download result (json) of successful imports
  • Cancel (Revoke) a running import (parallel or non-parallel)
  • Manual option to refresh list
  • Filter by Date (started)
  • Search by id/queue/status
• Queue New Import
  • Submit JSON (no validation for now)
  • Submit JSON File URL
  • Upload JSON File (upto 95MB)
  • Parallel Mode option (default off) (default parallel workers assigned 2)
  • (optional) Assign Custom Queue option
• Imports Screen Navigation
  • Add option in User Options below "My Profile"
  • Label for option "Bulk Imports"
  • Only available for logged in user
  • User can view their own imports only

[jamlung-ri]

Future ideas from our Arch call today:

• Add explanatory tooltips to "Update Existing" and "Parallel" checkboxes
• Navigate Existing Imports list using text search and date range

[snyaggarwal]

@paynejd

Notes:

1. You can now define a custom queue for bulk import parallel mode. Add queuing feature to all bulk import methods #704 is Done.
2. Custom queue is only accepted for non-admin users. Admin users always get the "priority" queue.
3. For parallel import -- right now, by default from Web you will get 2 workers (hardcoded).
   • We can make users enter this number, but then we will have to put some validation on the max/min number of workers.
   • We can assign ourselves by analyzing the JSON based on line count (later?)
   • We can just keep this number to a hardcoded value, like 2, (change by env?) assuming users wont queue big bulk-imports.

[jamlung-ri]

A couple of small things I am noticing during testing:

• If I run a successful import but did not have access to the orgs, the import itself is shown as successful but all of the concepts were skipped. I wonder if there is a way we can reflect this in the status of the import?
• "Download" button should be replaced with "Download Report" or something more clear.
• The date button has a tooltip "Updated since", but the behavior seems to be more like "Updated on". If I select yesterday, it won't show today's imports.

[snyaggarwal]

@jamlung-ri

• Date filter is to just filter by a specific date. I have updated the tooltip.
• "Download" is now "Download Report"
• In order to reflect that in the status we will have to do a lot more. For now, downloading the report and analyzing manually is the only option. Let me know if this is important, I will create a ticket for this.

[paynejd]

this is awesome!