Skip to content

functions specialized in comparing and merging arrays of elements signed by unique set of IDs

License

Notifications You must be signed in to change notification settings

knee-cola/array-merge-by-id

Repository files navigation

What's this?

This is a set of functions specialized in comparing and merging arrays of elements signed by unique set of IDs. This is typically used in scenario in which client-side data-sets with changes coming from the an SQL server (server > client replication).

How to install?

Simply open up a terminal, go to your project directory and run:

npm i --save array-merge-by-id

Try it before you buy it

You can try out all the examples (listed below) in your browser at RunKit.

Functions by category

Comparing, merging, linking elements of two arrays:

  • mergeA - merges new/changed elements into an existing array
  • compareA - compares elements of two arrays and returns an object containing common elements and differences
  • linkA - extends each child array element with a reference to it's parent element
  • eachPair - calls a callback method for each matched elements of provided arrays

Searching & filtering:

  • filterByKeys - extract all the array elements which match the given key values (or are indicated by a comparer function)
  • findFirstById - returns the first matched element of the given type
  • indexOf - returns index of first matching element in the given array
  • uniqueA - copies unique elements from source to a new array, which is then returned

Modifying, sorting, adding, removing elements of a single array:

  • concat - does an in-place concatination of elements of the source array at the end of target array
  • clear - removes all the elements of the given array
  • purgeA - removes elements indicated by a hit list from the provided array
  • overwrite - removes all the elements of target and replaces them with elements from source
  • sortOn - sorts the given array based on the given key name array (or comparer function)

Other helper functions:

  • compileC - compiles and returns a function which compares two data elements and detects which comes before which in an orderd list
  • stringifyIDs - creates CSV containing values of all the keys ending with "ID". It's usefull for debugging

TypeScript typings

This lib was written in TypeScript, which is good news for people using TypeScript - all the typing information is included in the build (don't worry - it can still be used in vanilla JavaScript ... etc).

Functions

compareA(leftA, rightA, key_columns, config)ArrayDiffResult

Compares elements of two arrays and returns an object containing common elements and differences (see comparisson results object).

filterByKeys(aSearch, key_columns, key_values, findFirstOnly)Array

Extract all the array elements which match the given key values (or are indicated by a comparer function)

findFirstById(aSearch, key_columns, key_values)Object

Returns the first matched element of the given type

indexOf(aSearch, key_columns, key_values)number

Returns index of first matching element in the given array

linkA(parentA, childA, key_columns, linkName, config)ArrayLinkResult

Extends each child array element with a reference to it's parent element (in parentA array). Optionally it can extend parane element with a map of it's children.

mergeA(currData, newData, key_columns, config)ArrayDiffResult

Merges new/changed elements into an existing array

purgeA(aTarget, aHitList, key_columns, config)Array

Removes elements indicated by a hit list from the provided array

uniqueA(source, key_columns, config)Array

Copies unique elements from source to a new array, which is then returned

compileC(key_columns)function

Compiles and returns a function which compares two data elements and detects which comes before which in an orderd list. The compiled function expects the compared values to be numeric

eachPair(leftA, rightA, key_columns, callbackFn, config)ArrayDiffResult

Calls a callback method for each matched elements of provided arrays

sortOn(source, key_columns)Array

Sorts the given array based on the given key name array (or comparer function)

clear(target)Array

Removes all the elements of the given array

overwrite(target, source)Array

removes all the elements of target and replaces them with elements from source

concat(target, source)Array

Does an in-place concatination of elements of the source array at the end of target array

stringifyIDs(data)string

Creates CSV containing values of all the keys ending with "ID". It's usefull for debugging

compareA(leftA, rightA, key_columns, config) ⇒ ArrayDiffResult

Compares elements of two arrays and returns an object containing common elements and differences (see comparisson results object).

Kind: global function
Returns: ArrayDiffResult - comparisson results object (see ArrayDiffResult)

Param Type Description
leftA Array first array be compared
rightA Array second array be compared
key_columns CompareBy definition on how elements of two arrays should be compared (see CompareBy)
config ArrayDiffConfig (optional) additional config parameters (see ArrayDiffConfig)

Example

let leftA = [
  {cityID:1, cityName:'New York', weather:"windy"},
  {cityID:2, cityName:'London',   weather:"raining"}
];

let rightA = [
  {cityID:2, cityName:'London', weather:"thunderstorm"},
  {cityID:3, cityName:'Moscow', weather:"snowing"}
];

let diff = compareA(leftA, rightA, ["cityID"]);

// statment below will print:
// {
//   leftDiff: [{cityID:1, cityName:'New York', weather:"windy"}],        <= New York doesn't exist in the `rightA`
//   leftCommon: [{cityID:2, cityName:'London', weather:"raining"}],      <= London does exist in `rightA`
//   rightDiff: [{cityID:3, cityName:'Moscow', weather:"snowing"}],       <= Moscow exists in `leftA`
//   rightCommon: [{cityID:2, cityName:'London', weather:"thunderstorm"}] <= London exists in `leftA`
// }
console.dir(diff);

filterByKeys(aSearch, key_columns, key_values, findFirstOnly) ⇒ Array

Extract all the array elements which match the given key values (or are indicated by a comparer function)

Kind: global function
Returns: Array - array of matched elements

Param Type Description
aSearch Array array to be searched
key_columns CompareBy definition on how elements of two arrays should be compared (see CompareBy)
key_values Map (optional) an object which should be matched with an element from aSearch array - it's optional because key_columns param can contain a function which doesn't need it
findFirstOnly boolean (optional) should only the first matched element be returned (defaults to false)

Example

let streets = [{cityID:22, streetID:1}, {cityID:44, streetID:2}, {cityID:22, streetID:3}];

let myCityStreets = filterByKeys(streets, ['cityID'], {cityID:22});

console.dir(myCityStreets); // will output [{cityID:22, streetID:1}, {cityID:22, streetID:3}]

findFirstById(aSearch, key_columns, key_values) ⇒ Object

Returns the first matched element of the given type

Kind: global function
Returns: Object - matched array element

Param Type Description
aSearch Array array to be searched
key_columns CompareBy definition on how elements of two arrays should be compared (see CompareBy)
key_values Map (optional) an object which should be matched with an element from aSearch array - it's optional because key_columns param can contain a function which doesn't need it

Example

let streets = [{cityID:22, streetID:1}, {cityID:44, streetID:2}, {cityID:22, streetID:3}];

let myStreet = findFirstById(streets, ['cityID','streetID'], {cityID:44, streetID:2});

console.dir(myStreet); // will output {cityID:44, streetID:2}

indexOf(aSearch, key_columns, key_values) ⇒ number

Returns index of first matching element in the given array

Kind: global function
Returns: number - element index

Param Type Description
aSearch Array array to be searched
key_columns CompareBy definition on how elements of two arrays should be compared (see CompareBy)
key_values Map (optional) an object which should be matched with an element from aSearch array - it's optional because key_columns param can contain a function which doesn't need it

Example

let streets = [{cityID:22, streetID:1}, {cityID:44, streetID:2}, {cityID:22, streetID:3}];

let streetIndex = findFirstById(streets, ['cityID','streetID'], {cityID:44, streetID:2});

console.log(streetIndex); // will print 1

linkA(parentA, childA, key_columns, linkName, config) ⇒ ArrayLinkResult

Extends each child array element with a reference to it's parent element (in parentA array). Optionally it can extend parane element with a map of it's children.

Kind: global function
Returns: ArrayLinkResult - linking results object(see ArrayLinkResult)

Param Type Default Description
parentA Array array containting parent elements
childA Array array containing child elements
key_columns CompareBy definition on how elements of two arrays should be compared (see CompareBy)
linkName string "parent" (optional) name of the property which should be assigned a reference to parent element (defaults to 'parent')
config ArrayDiffConfig (optional) additional config parameters (see ArrayDiffConfig)

Example

let cities = [ {cityID:22, cityName:'New York'}, {cityID:44, cityName:'London'} ];
let streets = [{cityID:22, streetID:1}, {cityID:44, streetID:2}, {cityID:22, streetID:3}];

let result = linkA(cities, streets, ['cityID'], 'city')

console.dir(result); // will output [ {cityID:22,streetID:1,city:<reference to New York>},
                     //               {cityID:44,streetID:2,city:<reference to London>},
                     //               {cityID:22,streetID:3,city:<reference to New York>}]

mergeA(currData, newData, key_columns, config) ⇒ ArrayDiffResult

Merges new/changed elements into an existing array

Kind: global function
Returns: ArrayDiffResult - comparisson results object (see ArrayDiffResult)

Param Type Description
currData Array an array of "current" data elements
newData Array an array of changes and new data elements
key_columns CompareBy definition on how elements of two arrays should be compared (see CompareBy)
config ArrayDiffConfig (optional) additional config parameters (see ArrayDiffConfig)

Example

let currData = [
  {cityID:1, cityName:'New York'},
  {cityID:2, cityName:'Londonnnnn'}
];

let newData = [
  {cityID:2, cityName:'London'}, // London is fixed
  {cityID:3, cityName:'Rome' }   // Rome is added
];

// function which applies changes to an existing element
let mergeFn = (element, changes) => { element.cityName = changes.cityName; };

let result = mergeA(currData, newData, ['cityID'], { callbackFn: mergeFn });

console.dir(currData); // will print [ {cityID:1, cityName:'New York'},
                       //              {cityID:2, cityName:'London'},
                       //              {cityID:3, cityName:'Rome' } ];

purgeA(aTarget, aHitList, key_columns, config) ⇒ Array

Removes elements indicated by a hit list from the provided array

Kind: global function
Returns: Array - an array of removed elements

Param Type Description
aTarget Array array to be purged
aHitList Array hit list - indicates which element from aTarget should be removed
key_columns CompareBy definition on how elements of two arrays should be compared (see CompareBy)
config ArrayPurgeConfig (optional) additional config parameters (see ArrayPurgeConfig)

Example

let targetA = [
  {cityID:1, cityName:'New York'},
  {cityID:2, cityName:'London'},
  {cityID:3, cityName:'Rome' }
];

let hitList = [ { cityID: 1 }, { cityID: 3 } ];

let elFreq = [];

let result = purgeA(targetA, hitList, ['cityID'] { elFreq: elFreq });

console.dir(targetA); // will print [ {cityID:1, cityName:'New York'} ];
console.dir(elFreq); // will print [1, 2] - it means that the first array
                     // element was found once, while the second twice

uniqueA(source, key_columns, config) ⇒ Array

Copies unique elements from source to a new array, which is then returned

Kind: global function
Returns: Array - array of unique elements

Param Type Description
source Array source array - which may contain duplicates
key_columns CompareBy definition on how elements of two arrays should be compared (see CompareBy)
config ArrayUniqueConfig (optional) additional config parameters (see ArrayUniqueConfig)

Example

let source = [ {cityID:1, cityName:'New York'}, {cityID:2, cityName:'London'}, {cityID:2, cityName:'London'} ];

let result = uniqueA(source, ["cityID"]);

console.dir(result); // will print [ {cityID:1, cityName:'New York'}, {cityID:2, cityName:'London'} ]

compileC(key_columns) ⇒ function

Compiles and returns a function which compares two data elements and detects which comes before which in an orderd list. The compiled function expects the compared values to be numeric

Kind: global function
Returns: function - compiled function

Param Type Description
key_columns CompareBy definition on how elements of two arrays should be compared (see CompareBy)

Example

// function for an ASCENDING order
let ascFn = compileC('cityID','streetID');

// will return 0
ascFn({cityID:1, streetID:1}, {cityID:1, streetID:1});

// will return 1
ascFn({cityID:1, streetID:1}, {cityID:1, streetID:2});

// will return -1
ascFn({cityID:2, streetID:1}, {cityID:1, streetID:2});

// function for an DESCENDING order
let descFn = compileC('cityID:desc','streetID:desc');

// will return 0
descFn({cityID:1, streetID:1}, {cityID:1, streetID:1});

// will return -1
descFn({cityID:1, streetID:1}, {cityID:1, streetID:2});

// will return 1
descFn({cityID:2, streetID:1}, {cityID:1, streetID:2});

eachPair(leftA, rightA, key_columns, callbackFn, config) ⇒ ArrayDiffResult

Calls a callback method for each matched elements of provided arrays

Kind: global function
Returns: ArrayDiffResult - comparisson results object

Param Type Description
leftA Array first array of elements
rightA Array second array of elements
key_columns CompareBy definition on how elements of two arrays should be compared (see CompareBy)
callbackFn function function to be called for each of mathced element pairs
config ArrayDiffConfig (optional) additional config parameters (see ArrayDiffConfig)

Example

let cities = [ {cityID:22, name:'New York'}, {cityID:44, name:'London'} ];

let streets = [
  {cityID:22, streetID:1, name:'Elm Street'},
  {cityID:22, streetID:3, name:'Wall St'},
  {cityID:44, streetID:2, name:'Downing St'}
];

let callbackFn = (city, street) => {
  console.log(street.name + ' ' + city.name)
}

eachPair(cities, streets, ["cityID"], callbackFn);

// The following will be printed to console:
// Elm Street in New York
// Wall St in New York
// Downing St in London

sortOn(source, key_columns) ⇒ Array

Sorts the given array based on the given key name array (or comparer function)

Kind: global function
Returns: Array - array passed via source param

Param Type Description
source Array array to be sorted
key_columns CompareBy definition on how elements of two arrays should be compared (see CompareBy)

Example

let source = [{cityID:2},{cityID:3},{cityID:1}];

// sorting in ASCENDING order
sortOn(source, ["cityID"]);

console.log(source); // will print [{cityID:1},{cityID:2},{cityID:3}];

// sorting in DESCENDING order
sortOn(source, ["cityID:desc"]);

let streets = [
   {cityID:22, streetID:1, name:'Elm Street'},
   {cityID:44, streetID:2, name:'Downing St'},
   {cityID:22, streetID:3, name:'Wall St'}
];

// using multiple keys
sortOn(source, ["cityID", "streetID:desc"]);

clear(target) ⇒ Array

Removes all the elements of the given array

Kind: global function
Returns: Array - array passed via target param

Param Type Description
target Array array to be cleared

Example

let source = [{cityID:2},{cityID:3},{cityID:1}];

clear(source);

console.dir(source); // will print []

overwrite(target, source) ⇒ Array

removes all the elements of target and replaces them with elements from source

Kind: global function
Returns: Array - array passed via target param

Param Type Description
target Array array to be overwritten
source Array array who's elements are to be places in target

Example

let target = [{cityID:100},{cityID:200}];
let source = [{cityID:2},{cityID:3},{cityID:1}];

overwrite(target, source);

console.dir(target); // will print [{cityID:2},{cityID:3},{cityID:1}]

concat(target, source) ⇒ Array

Does an in-place concatination of elements of the source array at the end of target array

Kind: global function
Returns: Array - array passed via target param

Param Type Description
target Array array which should receive new elements
source Array array of elements which should be added at the end of the target array

Example

let target = [{cityID:100},{cityID:200}];
let source = [{cityID:2},{cityID:3},{cityID:1}];

concat(target, source);

console.dir(target); // will print [{cityID:100},{cityID:200},{cityID:2},{cityID:3},{cityID:1}]

stringifyIDs(data) ⇒ string

Creates CSV containing values of all the keys ending with "ID". It's usefull for debugging

Kind: global function
Returns: string - CSV of key values

Param Type Description
data Object object or array to be converted to CSV

Example

let obj = { cityID: 11, streetID: 22, streetName: 'Elm Street' };

let result = stringifyIDs(obj);

console.log(result); // will print "cityID:11,streetID:22"

Param types

CompareBy

Functions which compare array elements need to be instructed how two elements can be compared. This can be done in two ways:

  • by passing an array of ID property names, which should be compared to determin the relation of the two objects
  • by passing a comparer function, which receives tow elements and returns a numeric value indicating the relation of the objects

If an array of property names is passed, a comparer function will be compiled automatically (via compileC) function.

The following snippet shows how comparisson can be defined via a comparer function:

let leftA = [
  {cityID:1, cityName:'New York', weather:"windy"},
  {cityID:2, cityName:'London',   weather:"raining"}
];

let rightA = [
  {cityID:2, cityName:'London', weather:"thunderstorm"},
  {cityID:3, cityName:'Moscow', weather:"snowing"}
];

const comparerFn = (leftEl, rightEl) => {
    if(leftEl.cityID===rightEl.cityID) {
        return(0);
    }
    if(leftEl.cityID<rightEl.cityID) {
        return(1);
    }
    return(0);
}

let diff = compareA(leftA, rightA, comparerFn);

The previous example can be re-written to define comparrison via property name array instead of comparer function:

let leftA = [
  {cityID:1, cityName:'New York', weather:"windy"},
  {cityID:2, cityName:'London',   weather:"raining"}
];

let rightA = [
  {cityID:2, cityName:'London', weather:"thunderstorm"},
  {cityID:3, cityName:'Moscow', weather:"snowing"}
];

let diff = compareA(leftA, rightA, ["cityID"]);

Descending order

If a ID name array is passed as key_columns params, the compiled function will compare elements in ascending order. We can change this behaviour by appending :desc to a ID name ... like so: ['cityID:desc','streetID:desc']

See an example given for the sortOn function.

ArrayDiffConfig

We can modify the way the functions work, by providing an config object. All the options in the config object are optional.

Here's a list of available options:

  • sortLeftBy - how should the leftA array be sorted
  • sortRightBy - how should the rightA be sorted
  • skipSort - set it to true if arrays are not to be sorted
  • unique - set it to true if are all the array elements unique - it speeds up the algorithm
  • linkName - property assigned to the rightA elements, which should be pointing to the matching element in the leftA
  • mapName - property assigned to the leftA elements, containig array of all the matched elemens from the rightA
  • callbackFn - a callback function, which should be called for each of the matched element pairs

sortLeftBy and sortRightBy

Description: defines how the arrays passed to the function should be sorted

Defaults to: value passed as key_columns param

Expected value: we can pass a function or an array of ID param names (see CompareBy)

In order to be more efficient, functions which rely on comparing array elements will sort both of the given arrays.

By default the functions use key_columns parameter to sort the arrays.

This can be overriden by specifying a dedicated sorting order for each of the two arrays:

  • config.sortLeftBy = defines how the left array should be sorted (passed as the first param)
  • config.sortRightBy = defines how the right array should be sorted (passed as the second param)

Sorting of an array can be disabled by assigning null to corresponding sort config param:

{
    sortLeftBy: null // don't sort the left array
}

ArrayPurgeConfig

This config si very similar to ArrayDiffConfig param type. The following params are the same as in ArrayDiffConfig:

  • sortLeftBy
  • sortRightBy
  • skipSort

The following params are unique to this param type:

  • sortBy = how should both arrays be sorted
  • mapRemoved - flag indicating should removed elements be mapped and returned
  • matchMulti - can an element from the hit list array be matched with multiple elements from the target array (defaults to false)

ArrayUniqueConfig

  • skipSort - set it to true if arrays are not to be sorted
  • elFreq - output param - an array in which element frequency is to be recorded (see the example given in the uniqueA method description)

Return Types

ArrayDiffResult

ArrayDiffResult contains results of comparing two arrays. It has the following structure:

{
    // an array of elements from left array (`leftA` param), which have not
    // been matched with any of the elements of the right array (`rightA` param)
    leftDiff:Array,
    // an array of elements from left array (`leftA` param),
    // which have been matched with at leas one element of the right array (`rightA` param)
    leftCommon:Array,
    // an array of elements from right array (`rightA` param), which have not
    // been matched with any of the elements of the left array (`leftA` param)
    rightCommon:Array,
    // an array of elements from right array (`rightA` param),
    // which have been matched with at leas one element of the left array (`leftA` param)
    rightDiff:Array
}

ArrayLinkResult

ArrayLinkResult contains results of linking two arrays. It has the following structure:

{
    // parent elements with no matched child elements
    childless:Array,
    // parent elements with matching children
    parents:Array,
    // child elements with matching parent
    children:Array,
    // child elements without a matching parent
    orphans:Array
}

License

MIT License, http://www.opensource.org/licenses/MIT

About

functions specialized in comparing and merging arrays of elements signed by unique set of IDs

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published