Workflow to add many entities at once

[pudo]

Users have requested that it should be easier to add many new entities to a collection at once. Our users often have existing search lists, which come in two formats:

• simple, line-delimited name lists used during the Panama Papers for bulk search (without further details, such as jurisdictions, birth dates etc.)
• multi-sheet Excel files created as "personal databases" for an investigation. These will typically have one sheet each for People, Companies, Lawyers, and Links between these.

With this ticket, we'd like to prototype a simple solution that allows users to submit these lists and turn them into Aleph collections. This could take the form of a large textarea, a CSV importer, or a CSV and Excel importer. If structured data (CSV) is to be imported, the users would probably need a mapping screen, on which they can describe the functionality of each column in the source data.

Some random questions on this:

• Do we want to implement this as a JavaScript-only component? There are good JS CSV parsers, the lists are typically small (never more than thousands of entries) and we have an existing entity creation API?
• Can we do something to de-duplicate new entities that come in against those that already exist in the collection? Should this be it's own screen?

Relates to #64, which is the inverse -- exporting entity lists as lead lists to be used in other search engines.

cc: @stas - I will provide you with samples of both import formats.

[stas]

Do we want to implement this as a JavaScript-only component? There are good JS CSV parsers, the lists are typically small (never more than thousands of entries) and we have an existing entity creation API?

Agree on this, this should also work with minor changes on the back-end side. Also, some of the libraries have decent copy/paste support for excel and similar formats. A good example of a library could be https://github.com/handsontable/ngHandsontable
Any major concerns here?

Can we do something to de-duplicate new entities that come in against those that already exist in the collection? Should this be it's own screen?

Well, we already have a per-entity Possible Duplicates screen. I suggest we either indicate the duplication score or come-up with a separate screen with bulk management support (how often and how big of a problem is this normally?). I'd lean more towards option B, except if this is not a major feature Aleph needs to provide (need your opinion here).