Field monitoring

[stephencarr]

Configuring the app to monitor only specific fields doesn't seem to work. Even though I have specified a 'fields' array, search still returns results as if all fields are being monitored. Perhaps I have misunderstood how this is supposed to work?

[katowulf]

Correct. It assumes we are monitoring records of data that will be added, deleted, and changed over time.

Can you explain the use case for monitoring a single field? Can't you just field.match(//) against it if your needs are this simple?

[stephencarr]

My table has a number of fields, I only want to search against specific fields. I have limited experience with elastic search so pardon my ignorance if I have this back-to-front.

[katowulf]

Specify the fields you want to search in config.js using the fields array

[stephencarr]

Perhaps my original explanation was a little rushed, but this is what I had done and it doesn't seem to work. With config.js specifying fields: ['x', 'y', 'z'] I still get results from matches on fields 'a', 'b', and 'c'.

[katowulf]

That could be a bug. Would it be possible to share your config.js, data structure, and client code?

[stephencarr]

I think Config is the only file I have changed from the cloned code. I am really only just testing this out.

Config.js

/**
 * This config file is provided as a convenience for development. You can either
 * set the environment variables on your server or modify the values here.
 *
 * At a minimum, you must set FB_URL and Paths to Monitor. Everything else is optional, assuming your
 * ElasticSearch server is at localhost:9200.
 */

/** Firebase Settings
 ***************************************************/

// Your Firebase instance where we will listen and write search results
exports.FB_URL = 'https://economatic.firebaseio.com/';

// Either your Firebase secret or a token you create with no expiry, used to authenticate
// To Firebase and access search data.
exports.FB_TOKEN = process.env.FB_TOKEN || null;

// The path in your Firebase where clients will write search requests
exports.FB_REQ = process.env.FB_REQ || 'search/request';

// The path in your Firebase where this app will write the results
exports.FB_RES = process.env.FB_RES || 'search/response';


/** ElasticSearch Settings
 *********************************************/

if (process.env.BONSAI_URL) {
    processBonsaiUrl(exports, process.env.BONSAI_URL);
} else {
    // ElasticSearch server's host URL
    exports.ES_HOST = process.env.ES_HOST || 'localhost';

    // ElasticSearch server's host port
    exports.ES_PORT = process.env.ES_PORT || '9200';

    // ElasticSearch username for http auth
    exports.ES_USER = process.env.ES_USER || null;

    // ElasticSearch password for http auth
    exports.ES_PASS = process.env.ES_PASS || null;
}


/** Paths to Monitor
 *
 * Each path can have these keys:
 * {string}   path:    [required] the Firebase path to be monitored, for example, `users/profiles`
 *                     would monitor https://<instance>.firebaseio.com/users/profiles
 * {string}   index:   [required] the name of the ES index to write data into
 * {string}   type:    [required] name of the ES object type this document will be stored as
 * {Array}    fields:  list of fields to be monitored (defaults to all fields)
 * {Function} filter:  if provided, only records that return true are indexed
 * {Function} parser:  if provided, the results of this function are passed to ES, rather than the raw data (fields is ignored if this is used)
 ****************************************************/

exports.paths = [{
    path: "cars",
    index: "firebase",
    type: "car",
    fields: ['MANUFACTURER', 'MODEL', 'YEAR']
}];


/** Config Options
 ***************************************************/

// How often should the script remove unclaimed search results? probably just leave this alone
exports.CLEANUP_INTERVAL =
    process.env.NODE_ENV === 'production' ?
    3600 * 1000 /* once an hour */ :
    60 * 1000 /* once a minute */ ;

function processBonsaiUrl(exports, url) {
    var matches = url.match(/^https?:\/\/([^:]+):([^@]+)@([^/]+)\/?$/);
    exports.ES_HOST = matches[3];
    exports.ES_PORT = 80;
    exports.ES_USER = matches[1];
    exports.ES_PASS = matches[2];
    console.log('Configured using BONSAI_URL environment variable', url, exports);
}

[katowulf]

So all we would need is a short sample of your data and the client-side code you're using to read the results and determine that it searches on all fields. Then I should be able to pinpoint why you're not getting the desired results. Feel free to email data to wulf at firebase or just sanitize sensitive fields; a few records should suffice.

[stephencarr]

Data emailed to you, thanks.

[jwall]

I have the same issue. No matter what I put in the fields array be it

exports.paths = [
                 {
                     path:  "profiles",
                     index: "firebase",
                     type:  "profile",
                     fields: ['foobar']
                 }
                 ];

I always get back a match on all fields when I would think if I don't have any key of 'foobar', I would get no matches.

[katowulf]

Thanks for the update; will have a look at this soon.

[dustinboss]

So, I dug into this, and it looks like the "fields" param in export.paths isn't referenced anywhere else in Flashlight. The logical place would be in PathMonitor at line 6 (where the Firebase ref is being constructed), but I think that even that would be unnecessary.

Perhaps just eliminate "fields" as a parameter altogether, and just create more complete paths. Example: path: "profiles" could be path: "profiles/foobar"

In either case, having "fields" in the config.js comments is confusing, since it doesn't do anything as it currently stands.

[katowulf]

Fixed in latest head branch. Thanks gents!

[stephencarr]

w00t! 👍