pytx: Searching for a ThreatDescriptor shows 'privacy_type': None

[lolnate]

I added a ThreatDescriptor with a privacy_type of "HAS_PRIVACY_GROUP", but searching for it in pytx is returning "None" as the privacy_type. Just to note, the indicator is not sensitive. I was just testing. Details below.

Using pytx, I added the following ThreatDescriptor:

# Let's try to submit a new Threat Descriptor
from pytx import ThreatDescriptor
from pytx.vocabulary import ThreatDescriptor as tdv
from pytx.vocabulary import Types, Precision, PrivacyType, ReviewStatus, Severity, ShareLevel, Status

params = {
    tdv.INDICATOR : 'http://212.154.211.81/giz.exe',
    tdv.TYPE : Types.URI,
    tdv.CONFIDENCE : 75,
    tdv.DESCRIPTION : 'Ransomware download URL',
    tdv.PRECISION : Precision.MEDIUM,
    tdv.PRIVACY_MEMBERS : '1125937020771155', # CatFanciers ID
    tdv.PRIVACY_TYPE : PrivacyType.HAS_PRIVACY_GROUP,
    tdv.REVIEW_STATUS : ReviewStatus.REVIEWED_AUTOMATICALLY,
    tdv.SEVERITY : Severity.SUSPICIOUS,
    tdv.SHARE_LEVEL : ShareLevel.AMBER,
    tdv.STATUS : Status.MALICIOUS,
    tdv.TAGS : 'sage,ransomware,http_request,malware',
}

result = ThreatDescriptor.new(params=params)
print(result)

The following response returned: {'id': '1447134161986113', 'success': True}

Then I took a look at the indicator directly in my browser with: https://graph.facebook.com/v2.8/1447134161986113/?access_token=[REDACTED]

It showed what I would expect:

{
   "added_on": "2017-02-23T16:38:27+0000",
   "id": "1447134161986113",
   "indicator": {
      "indicator": "http://212.154.211.81/giz.exe",
      "type": "URI",
      "id": "1447134155319447"
   },
   "owner": {
      "id": "1678314142420566",
      "email": "nlhausrath\u0040ashland.com",
      "name": "Ashland CIRT"
   },
   "type": "URI",
   "raw_indicator": "http://212.154.211.81/giz.exe",
   "description": "Ransomware download URL",
   "status": "MALICIOUS",
   "privacy_type": "HAS_PRIVACY_GROUP",
   "share_level": "AMBER"
}

However, when I did the following, privacy_type was set to 'None':

from pytx import ThreatDescriptor

results = ThreatDescriptor.objects(
    text='giz.exe',
    owner='1678314142420566', # me
)

for result in results:
    print(result.to_dict())

The following was printed:

{'privacy_members': None, 'severity': 'SUSPICIOUS', 'owner': {'id': '1678314142420566', 'name': 'Ashland CIRT', 'email': 'nlhausrath@ashland.com'}, 'privacy_type': None, 'source_uri': '', 'id': '1447134161986113', 'share_level': 'AMBER', 'expired_on': None, 'precision': 'MEDIUM', 'review_status': 'REVIEWED_AUTOMATICALLY', 'metadata': None, 'indicator': {'type': 'URI', 'id': '1447134155319447', 'indicator': 'http://212.154.211.81/giz.exe'}, 'status': 'MALICIOUS', 'my_reactions': None, 'raw_indicator': 'http://212.154.211.81/giz.exe', 'type': 'URI', 'description': 'Ransomware download URL', 'added_on': '2017-02-23T16:38:27+0000', 'last_updated': '2017-02-23T16:38:28+0000', 'tags': {'data': [{'id': '1382721905133632', 'text': 'http_request'}, {'id': '1375757795798370', 'text': 'ransomware'}, {'id': '1318516441499594', 'text': 'malware'}, {'id': '595090370615714', 'text': 'sage'}]}, 'confidence': 75}

Am I doing something wrong or is this the wrong expectation? Thanks!

[mgoffin]

You aren't doing anything wrong per-se :) With a recent update we will only return the contents of _default_fields:

https://github.com/facebook/ThreatExchange/blob/master/pytx/pytx/threat_descriptor.py#L36

With nothing to set to the attribute coming back from the server, it will be shown as None. You can adjust your query to look like this to get the full contents:

from pytx import ThreatDescriptor

results = ThreatDescriptor.objects(
    text='giz.exe',
    owner='1678314142420566', # me
    fields=ThreatDescriptor._fields
)

for result in results:
    print(result.to_dict())

This will force the fields ThreatExchange returns to be all of the available fields. Before the recent change I'm not sure if you'd get the privacy_type field either. You could check that by running a query in your browser. If it is returned by default, or if you feel like it would be really helpful to have this field returned all the time, we can update the _default_fields to include it!

[lolnate]

Perfect, thanks! It looks like 'privacy_members' is still None, but the privacy_type is now set correctly.

It would be useful to know which privacy group specifically shared the descriptor. I might trust some groups more than others. I could also generate metrics for contributions to private groups. Is there another approach I could use to find that information?

[mgoffin]

I think that's something that I've asked about in the past, but related to searching for the privacy_group that shared the content. If you used _fields and privacy_members is still None, it's because ThreatExchange isn't returning it. So I suspect that data isn't provided, but hopefully someone else can answer whether that should be the case or not.

My gut feeling is that it's not shared because Privacy Groups have a members_can_see attribute, but I don't know, if that's set to False, if that means they can't see the actual Privacy Group and/or it in the privacy_members field.

[jessek]

@mgoffin is correct. A person can only see the name, description, and members if the members_can_see is field true and the person is a member of the group.

[lolnate]

Thanks for the help. Is there a way to know if a particular ThreatDescriptor or other object is shared with a privacy group? members_can_see let's members view information about the privacy group, but it doesn't seem to show any attachment of a group to a descriptor. Some privacy groups might be more important to me than others, so it's a distinction I'd like to know.

If not, I suppose I could approach it a different way. For example, I could get a list of all members in a privacy group, then look for descriptors that have that member as the owner with a privacy type of HAS_PRIVACY_GROUP.

[mgoffin]

I don't believe so. I think this was related to my request to allow searching on the privacy_members field for the ID of the privacy group, so I could say "I'm a member of the Foo privacy group, show me all Descriptors shared out to the Foo privacy group".

I don't think anything came of that, but the thread was in the FB group if you can find it :)