tor_lookup pipeline function returns always false

[mudrunkar]

Expected Behavior

I tested the pipeline function tor_lookup with several tor exit node's addressess, but all I got was {"threat_indicated":false} response. Does it work for anyone?

Current Behavior

Only negative response {"threat_indicated":false} is returned as a result of tor lookup, there is no error message in the log.

Possible Solution

Steps to Reproduce (for bugs)

I tested the lookup function with the following IP addressess where I would expect positive tor lookup result:

but I got false in case of all of these IP addressess.

Pipeline function used:

rule "add_tor_lookup"
when 
    has_field("source_address")
then
    let tor_lookup = tor_lookup(to_string($message.source_address));
    set_field("tor_lookup", tor_lookup);
end 

Does the (now built-in) tor lookup plugin work for someone? Maybe I'm doing something wrong?

Context

Your Environment

• Graylog Version: 2.4.1
• Elasticsearch Version:
• MongoDB Version:
• Operating System:
• Browser version:

[joschi]

@mudrunkar Please provide the full pipeline rule calling the tor_lookup() function which you're using to enrich your messages.

[mudrunkar]

I use the following function:

rule "add_tor_lookup"
when 
    has_field("source_address")
then
    let tor_lookup = tor_lookup(to_string($message.source_address));
    set_field("tor_lookup", tor_lookup);
end

[swelcher]

I'm having a similar issue, but just by looking at your example and by looking at the examples for plugin, that tor_lookup in the set_field needs to look like set_field("tor_lookup", tor_lookup.threat_indicated);

[swelcher]

@joschi I've been looking at TorExitNodeLookupFunction and LookupResult and I'm probably overthinking/missing this but where does "lookupFunction" come from in this.lookupFunction.lookup(ip.trim())?

final LookupResult lookupResult = this.lookupFunction.lookup(ip.trim());

In a way it almost appears to bypass the if/else statement and falls to the default GenericLookupResult.False value.

    @Override
    public GenericLookupResult evaluate(FunctionArgs args, EvaluationContext context) {
        String ip = valueParam.required(args, context);
        if (ip == null) {
            LOG.error("NULL parameter passed to Tor exit node lookup.");
            return null;
        }

        LOG.debug("Running Tor exit node lookup for IP [{}].", ip);

        final LookupResult lookupResult = this.lookupFunction.lookup(ip.trim());
        if (lookupResult != null && !lookupResult.isEmpty()) {
            final Object value = lookupResult.singleValue();
            if (value instanceof Boolean) {
                return (Boolean) value ? GenericLookupResult.TRUE : GenericLookupResult.FALSE;
            }
            if (value instanceof String) {
                return Boolean.valueOf((String) value) ? GenericLookupResult.TRUE : GenericLookupResult.FALSE;
            }
        }

        return GenericLookupResult.FALSE;
}

[ion-storm]

I can confirm, tor_lookup always fails, also when you query the dataset it does not return true when found, it returns what looks like a hash

Tor failed lookup:

{
  "single_value": "D83665AF257FD05C4687897815233FD52A8E9829",
  "multi_value": {
    "node_ids": [
      "D83665AF257FD05C4687897815233FD52A8E9829"
    ]
  },
  "ttl": 9223372036854776000,
  "empty": false
}

Abuse.CH successful lookup
{
  "single_value": true,
  "multi_value": {
    "value": true
  },
  "ttl": 9223372036854776000,
  "empty": false
}

[ion-storm]

When querying TOR, it'll respond as false when not found, returns a hash when found.

[ion-storm]

Comparing both Tor and abuse.ch lookup functions it looks like this may need to be changed to

        final LookupResult lookupResult = this.lookupFunction.lookup(ip.trim());
        if (lookupResult != null && !lookupResult.isEmpty() && lookupResult.singleValue() != null) {
            if (lookupResult.singleValue() instanceof Boolean) {
                return (Boolean)lookupResult.singleValue() ? GenericLookupResult.TRUE : GenericLookupResult.FALSE;
            }
            if (lookupResult.singleValue() instanceof String) {
                return Boolean.valueOf((String) lookupResult.singleValue()) ? GenericLookupResult.TRUE : GenericLookupResult.FALSE;
            }
}

[stamfest]

This still is an issue in the latest 2.4.6 version (at least in the ubuntu package). The intended functionality is unusable in the current state.

[danotorrey]

Thank you for all of the details. We are investigating this issue.

[stamfest]

Too bad that a fix for this issue has not made it into 2.5. Is that really such a complex issue? Any pointer on where this should be fixed so we can take a shot at it?

[danotorrey]

Hi @stamfest,
My sincere apologies on the delayed response. Thank you for following up again.

We have been investigating the issue this week and are making good progress in understanding why this is failing. I expect to have more info very soon.

[danotorrey]

@stamfest @ion-storm @mudrunkar @swelcher We have confirmed that this issue is occurring due to a bug. The bug will be fixed in Graylog version 3.0, which will be released next month.

[dio99]

will it not be fixed in 2.4/2.5?
// Anders

[danotorrey]

Hi @mudrunkar @stamfest @ion-storm @dio99,
This will be fixed in Graylog version 3.0 coming out next month, but there is a solid workaround:

1. Verify that a Tor Exit Node Lookup Table is set up.
2. Add the following pipeline rule that uses the lookup table by it's name:

rule "Is from a Tor Exit Node: src_addr"
when 
    has_field("src_addr")
then
    let tor_lookup = lookup( "name-of-tor-exit-node-lookup-table", to_string($message.src_addr));
    set_field("from_tor_exit_node", is_not_null(join(tor_lookup.node_ids)));
end

See this docs page for general instructions for setting up a lookup table. Please note that you may need to also enable Tor Lookups in System > Configurations > Threat Intelligence Lookup Configuration > Tor Exit Nodes.

[dio99]

complains about function join in graylog version 2.5
//Anders