A mixin to provide bulk import functionallity for Loopback Models
JavaScript
Permalink
Failed to load latest commit information.
dist Release 1.0.27 Dec 6, 2016
src Release 1.0.27 Dec 6, 2016
test Release 1.0.6 Apr 11, 2016
.babelrc Initial Commit Apr 4, 2016
.eslintrc Pre-Release 1.0.0 Apr 5, 2016
.gitignore Initial Commit Apr 4, 2016
.travis.yml Initial Commit Apr 4, 2016
CHANGELOG.md Release 1.0.16 Jul 22, 2016
LICENSE.md Initial Commit Apr 4, 2016
README.md Release 1.0.6 Apr 11, 2016
gulpfile.babel.js Initial Commit Apr 4, 2016
package.json Release 1.0.27 Dec 6, 2016

README.md

NPM NPM

Loopback Import Mixin

This module is designed for the Strongloop Loopback framework. It provides bulk import functionallity to Models and Relations by uploading CSV files.

It is capable to impor bulk sets of data by creating Models and it Relationships, also provides the ability to update existing instances by modifying it properties if any changes in values are found.

It provides a Log mechanisim that will create history that includes the import process with specific warnings, errors and details for each row in the file.

INSTALL

  npm install --save loopback-import-mixin loopback-component-storage

MIXINSOURCES

With loopback-boot@v2.8.0 mixinSources have been implemented in a way which allows for loading this mixin without changes to the server.js file previously required.

Add the mixins property to your server/model-config.json like the following:

{
  "_meta": {
    "sources": [
      "loopback/common/models",
      "loopback/server/models",
      "../common/models",
      "./models"
    ],
    "mixins": [
      "loopback/common/mixins",
      "../node_modules/loopback-import-mixin/dist",
      "../common/mixins"
    ]
  }
}

SETUP

The loopback-import-mixin requires you to setup the following datasource:

// my_project/server/datasources.json
{
  "container": {
    "name": "container",
    "connector": "loopback-component-storage",
    "provider": "filesystem",
    "root": "tmp" //mkdir tmp on my_project root: e.g. my_project/tmp
  }
}

Also you will need to create the following models:

Model Datasource Requried Public Base Properties
ImportContainer container Yes false Model N/A
ImportLog db Yes true PersistedModel SEE BELOW
ImportLog Properties

The ImportLog Model will keep track of your imports for any of the models you implement the loopback-import-mixin

Property Type Requried Description
status string Yes Will keep log of the import status (Pending, Processing, Finished).
model string Yes Will keep log of the model name where the import was executed.
date date Yes Will keep log of the moment in which the import was executed.
notices ["object"] No Will keep log of all of the successfull import processes.
warnings ["object"] No Will keep log of all of the not expected behaviour, e.g. duplicates.
errors ["object"] No Will keep log of all the unrecoverable errors.

HINT: By implementing remoteHooks you can validate the imported data before is persisted or send email / push notifications when finished with notices, warnings and errors over specific rows after is persisted. e.g. Model.beforeRemote('import') or Model.afterRemote('import')

IMPORT MIXIN

This mixin creates a Remote Method called import that accepts a csv file and then forks a new process to import the data related to a model and possible many-to-many relationships.

EXAMPLE OF MIXIN CONFIGURATION

You can configure the loopback-import-mixin by mapping the CSV file column names with the model property names, also you can map the relationship with other currently existing instances:

"mixins": {
    "Import": [{
        "pk": "csvFileColumnPK",
        "method": "importSomething",
        "map": {
            "modelProperty1": "csvFileColumnName1",
            "modelProperty2": "csvFileColumnName2",
            "modelProperty3": "csvFileColumnName3",
            "modelProperty4": "csvFileColumnName4",
            // ...
        },
        "relations": {
            "modelRelation1": {
                "type": "link",
                "where": {
                    "relatedModelProperty": "csvFileColumnNameX"
                }
            },
            "modelRelation2": {
                "type": "create",
                "map": {
                    "relatedModelPropertyFoo": "csvFileColumnNameFoo",
                    "relatedModelPropertyBar": "csvFileColumnNameBar"
                }
            }
        }
    }]
}

The code defined above would create a localhost:3000/api/model/import endpoint with the ability to import models with properties 1...4 within the map section.

In this example, the relation names MUST correspond to an actual relationship name defined in the Model e.g. "relations": {"modelRelation1": {...}}. and the items inside the relation object works as a where statement pseudo code: add Model.modelRelation1 where relatedModelProperty = csvFileColumnNameX

The where statement is transparently passed to loopback, meaning you can use or & and operators as any regular loopback where query:

"mixins": {
    "Import": {
        "pk": "csvFileColumnPK",
        "map": {
            "modelProperty1": "csvFileColumnName1",
            // ...
        },
        "relations": {
            "modelRelation1": {
                "type": "link",
                "where:" {
                    "or": [
                        { "name": "csvFileColumnNameX" },
                        { "email": "csvFileColumnNameY" }
                    ]
                }
            }
        }
    }
}

LICENSE

MTI