Skip to content

Commit

Permalink
XSUP-27936 problem with regex (demisto#29613)
Browse files Browse the repository at this point in the history 
* failed test

* fix

* rn

* rn

* unit test

* ut

* validations

* fixed test and docker

* fix

* validation
  • Loading branch information
@eyalpalo @maimorag
eyalpalo authored and maimorag committed Sep 28, 2023
1 parent 7898981 commit 16f1817
Show file tree
Hide file tree
Showing 8 changed files with 237 additions and 64 deletions.
5 changes: 4 additions & 1 deletion Packs/Whois/.secrets-ignore
View file
Expand Up @@ -54,4 +54,7 @@ http://cscdbs.com
http://www.apnic.net
http://rdap.apnic.net
helpdesk@apnic.net
https://www.apnic.net
https://www.apnic.net
test@test.net
http://www.test.net
test@test.fr
7 changes: 1 addition & 6 deletions Packs/Whois/Integrations/Whois/Whois.py
View file
Expand Up @@ -7658,12 +7658,7 @@ def preprocess_regex(regex):
# nic.ir, individual - this is a nasty one.
"nic-hdl:\s+(?P<handle>.+)\norg:\s+(?P<organization>.+)\n(?:e-mail:\s+(?P<email>.+)\n)?(?:address:\s+(?P<street1>.+?)(?:,+ (?P<street2>.+?)(?:,+ (?P<street3>.+?)(?:,+ (?P<street4>.+?)(?:,+ (?P<street5>.+?)(?:,+ (?P<street6>.+?)(?:,+ (?P<street7>.+?))?)?)?)?)?)?, (?P<city>.+), (?P<state>.+), (?P<country>.+)\n)?(?:phone:\s+(?P<phone>.+)\n)?(?:fax-no:\s+(?P<fax>.+)\n)?",
# nic.ir, organization
"nic-hdl:\s*(?P<handle>.+)\ntype:\s*(?P<type>.+)\ncontact:\s*(?P<name>.+)\n(?:.+\n)*?(?:address:\s*(?P<street1>.+)\naddress:\s*(?P<street2>.+)\naddress:\s*(?P<street3>.+)\naddress:\s*(?P<country>.+)\n)?(?:phone:\s*(?P<phone>.+)\n)?(?:fax-no:\s*(?P<fax>.+)\n)?(?:.+\n)*?(?:e-mail:\s*(?P<email>.+)\n)?(?:.+\n)*?changed:\s*(?P<changedate>.*}).*\n",
# AFNIC madness without country field
"nic-hdl:\s*(?P<handle>.+)\ntype:\s*(?P<type>.+)\ncontact:\s*(?P<name>.+)\n(?:.+\n)*?(?:address:\s*(?P<street1>.+)\n)?(?:address:\s*(?P<street2>.+)\n)?(?:address:\s*(?P<street3>.+)\n)?(?:phone:\s*(?P<phone>.+)\n)?(?:fax-no:\s*(?P<fax>.+)\n)?(?:.+\n)*?(?:e-mail:\s*(?P<email>.+)\n)?(?:.+\n)*?changed:\s*(?P<changedate>.*).*\n",
# AFNIC madness any country -at all-
"nic-hdl:\s*(?P<handle>.+)\ntype:\s*(?P<type>.+)\ncontact:\s*(?P<name>.+)\n(?:.+\n)*?(?:address:\s*(?P<street1>.+)\n)?(?:address:\s*(?P<street2>.+)\n)?(?:address:\s*(?P<street3>.+)\n)?(?:address:\s*(?P<street4>.+)\n)?country:\s*(?P<country>.+)\n(?:phone:\s*(?P<phone>.+)\n)?(?:fax-no:\s*(?P<fax>.+)\n)?(?:.+\n)*?(?:e-mail:\s*(?P<email>.+)\n)?(?:.+\n)*?changed:\s*(?P<changedate>.+).*\n",
# AFNIC madness with country field
"nic-hdl:[ ]*(?P<handle>.*?)\ntype:[ ]*(?P<type>.*)\ncontact:[ ]*(?P<name>.*?)\n(?:.*\n)*?(?:(?:address:[ ]*(?P<street1>.*?)\n)(?:address:[ ]*(?P<street2>.*?)\n)?(?:address:[ ]*(?P<street3>.*)\n)?(?:address:[ ]*(?P<street4>.*)\n)?(?:country:[ ]*(?P<country>.*?)\n)?)(?:phone:[ ]*(?P<phone>.*?)\n)?(?:fax-no:[ ]*(?P<fax>.*?)\n)?(?:.*\n)*?(?:e-mail:[ ]*(?P<email>.*?)\n)?registrar:[ ]*(?P<registrar>.*?)\n(?:.*?\n)*?(?:changed:[ ]*(?P<changedate>.*?)\n)?"
]

organization_regexes = (
Expand Down
18 changes: 9 additions & 9 deletions Packs/Whois/Integrations/Whois/Whois.yml
View file
Expand Up @@ -160,10 +160,10 @@ script:
description: The organization of the domain administrator.
type: string
- contextPath: Domain.Whois.Administrator.postalcode
description: The postal code of the domain administrator
description: The postal code of the domain administrator.
type: string
- contextPath: Domain.Whois.Administrator.street
description: The street of the domain admin
description: The street of the domain admin.
type: string
- contextPath: Domain.Whois.Administrator.phone
description: The phone number of the domain administrator.
Expand Down Expand Up @@ -313,10 +313,10 @@ script:
description: The organization of the domain administrator.
type: string
- contextPath: Domain.Whois.Administrator.postalcode
description: The postal code of the domain administrator
description: The postal code of the domain administrator.
type: string
- contextPath: Domain.Whois.Administrator.street
description: The street of the domain admin
description: The street of the domain admin.
type: string
- contextPath: Domain.Whois.Administrator.phone
description: The phone number of the domain administrator.
Expand Down Expand Up @@ -484,7 +484,7 @@ script:
description: ASN allocation date in ISO 8601 format.
type: Date
- contextPath: Whois.IP.asn_description
description: The ASN description
description: The ASN description.
type: string
- contextPath: Whois.IP.asn_registry
description: ASN assigned regional internet registry.
Expand Down Expand Up @@ -547,10 +547,10 @@ script:
description: The RIR classification of a registered network.
type: string
- contextPath: Whois.IP.query
description: The IP address
description: The IP address.
type: string
- contextPath: IP.Address
description: IP address
description: IP address.
type: string
- contextPath: IP.ASN
description: 'The autonomous system name for the IP address, for example: "AS8948".'
Expand All @@ -565,7 +565,7 @@ script:
description: Indicators that are associated with the IP.
type: string
- contextPath: IP.feed_related_indicators.type
description: The type of the indicators that are associated with the IP
description: The type of the indicators that are associated with the IP.
type: string
- contextPath: DBotScore.Indicator
description: The indicator that was tested.
Expand All @@ -582,7 +582,7 @@ script:
- contextPath: DBotScore.Reliability
description: Reliability of the source providing the intelligence data.
type: String
dockerimage: demisto/ippysocks-py3:1.0.0.72626
dockerimage: demisto/ippysocks-py3:1.0.0.74506
runonce: false
script: '-'
subtype: python3
Expand Down
75 changes: 28 additions & 47 deletions Packs/Whois/Integrations/Whois/Whois_test.py
View file
Expand Up @@ -2,6 +2,7 @@
import pickle
import Whois
import demistomock as demisto
import pathlib
import pytest
import subprocess
import time
Expand Down Expand Up @@ -340,61 +341,24 @@ def test_whois_with_verbose(args, expected_res, mocker: MockerFixture):


def test_parse_nic_contact():
data = ["%%\n%% This is the AFNIC Whois server.\n%%\n%% complete date format : YYYY-MM-DDThh:mm:ssZ\n%% short date "
"format : DD/MM\n%% version : FRNIC-2.5\n%%\n%% Rights restricted by copyright.\n%% See "
"https://www.afnic.fr/en/products-and-services/services/whois/whois-special-notice/\n%%\n%% Use '-h' option"
"to obtain more information about this service.\n%%\n%% [1111 REQUEST] >> google.fr\n%%\n%% RL "
"Net [##########] - RL IP [#########.]\n%%\n\ndomain: google.fr\nstatus: ACTIVE\nhold: "
"NO\nholder-c: GIHU100-FRNIC\nadmin-c: GIHU101-FRNIC\ntech-c: MI3669-FRNIC\nzone-c: "
"NFC1-FRNIC\nnsl-id: NSL4386-FRNIC\nregistrar: MARKMONITOR Inc.\nExpiry Date: 2022-12-30T17:16"
":48Z\ncreated: 2000-07-26T22:00:00Z\nlast-update: 2022-08-17T16:39:47Z\nsource: FRNIC\n\nns-list:"
" NSL4386-FRNIC\nnserver: ns1.google.com\nnserver: ns2.google.com\nnserver: ns3.google.com\n"
"nserver: ns4.google.com\nsource: FRNIC\n\nregistrar: MARKMONITOR Inc.\ntype: Isp Option "
"\naddress: 2150 S. Bonito Way, Suite 150\naddress: ID 83642 MERIDIAN\ncountry: US\n"
"phone: +1 208 389 5740\nfax-no: +1 208 389 5771\ne-mail: registry.admin@markmonitor.com\n"
"website: http://www.markmonitor.com\nanonymous: NO\nregistered: 2002-01-10T12:00:00Z\nsource: "
"FRNIC\n\nnic-hdl: GIHU100-FRNIC\ntype: ORGANIZATION\ncontact: Google Ireland Holdings "
"Unlimited Company\naddress: Google Ireland Holdings Unlimited Company\naddress: 70 Sir John "
"Rogerson's Quay\naddress: 2 Dublin\naddress: Dublin\ncountry: IE\nphone: "
"+353.14361000\ne-mail: dns-admin@google.com\nregistrar: MARKMONITOR Inc.\nchanged: "
" 2018-03-02T18:03:31Z nic.fr\nanonymous: NO\nobsoleted: NO\neligstatus: not identified\n"
"reachstatus: not identified\nsource: FRNIC\n\nnic-hdl: GIHU101-FRNIC\ntype: ORGANIZATION"
"\ncontact: Google Ireland Holdings Unlimited Company\naddress: 70 Sir John Rogerson's Quay\n"
"address: 2 Dublin\ncountry: IE\nphone: +353.14361000\ne-mail: dns-admin@google.com\n"
"registrar: MARKMONITOR Inc.\nchanged: 2018-03-02T17:52:06Z nic.fr\nanonymous: NO\nobsoleted: "
" NO\neligstatus: not identified\nreachmedia: email\nreachstatus: ok\nreachsource: REGISTRAR\nreachdate: "
"2018-03-02T17:52:06Z\nsource: FRNIC\n\nnic-hdl: MI3669-FRNIC\ntype: ORGANIZATION\ncontact:"
"MarkMonitor Inc.\naddress: 2150 S. Bonito Way, Suite 150\naddress: 83642 Meridian\naddress: "
"ID\ncountry: US\nphone: +1.2083895740\nfax-no: +1.2083895771\ne-mail: "
"ccops@markmonitor"
".com\nregistrar: MARKMONITOR Inc.\nchanged: 2021-10-05T15:17:57Z nic.fr\nanonymous: NO\n"
"obsoleted: NO\neligstatus: ok\neligsource: REGISTRAR\neligdate: 2021-10-05T15:17:56Z\nreachmedia: "
"email\nreachstatus: ok\nreachsource: REGISTRAR\nreachdate: 2021-10-05T15:17:56Z\nsource: FRNIC\n\n"]
with open('./test_data/whois_response.txt') as f:
data = [f.read()]

res = Whois.parse_nic_contact(data)

expected = [{'handle': 'GIHU100-FRNIC', 'type': 'ORGANIZATION', 'name': 'Google Ireland Holdings Unlimited Company',
'street1': 'Google Ireland Holdings Unlimited Company', 'street2': "70 Sir John Rogerson's Quay",
'street3': '2 Dublin', 'phone': None, 'fax': None, 'email': None,
'changedate': '2018-03-02T18:03:31Z nic.fr'},
{'handle': 'GIHU101-FRNIC', 'type': 'ORGANIZATION', 'name': 'Google Ireland Holdings Unlimited Company',
'street1': "70 Sir John Rogerson's Quay", 'street2': '2 Dublin', 'street3': None, 'phone': None,
'fax': None, 'email': None, 'changedate': '2018-03-02T17:52:06Z nic.fr'},
'street3': '2 Dublin', 'country': 'IE', 'phone': '+353.14361000', 'fax': None, 'email': 'email@google.com',
'changedate': '2022-10-15T05:41:14.918179Z', 'registrar': 'MARKMONITOR Inc.', 'street4': None, },
{'handle': 'MI3669-FRNIC', 'type': 'ORGANIZATION', 'name': 'MarkMonitor Inc.',
'street1': '2150 S. Bonito Way, Suite 150', 'street2': '83642 Meridian', 'street3': 'ID',
'phone': None, 'fax': None, 'email': None, 'changedate': '2021-10-05T15:17:57Z nic.fr'},
{'handle': 'GIHU100-FRNIC', 'type': 'ORGANIZATION', 'name': 'Google Ireland Holdings Unlimited Company',
'street1': 'Google Ireland Holdings Unlimited Company', 'street2': "70 Sir John Rogerson's Quay",
'street3': '2 Dublin', 'street4': 'Dublin', 'country': 'IE', 'phone': '+353.14361000', 'fax': None,
'email': 'dns-admin@google.com', 'changedate': '2018-03-02T18:03:31Z nic.fr'},
'street1': '2150 S. Bonito Way, Suite 150', 'street2': '83642 Meridian', 'street3': None, 'street4': None,
'phone': '+1.2083895740', 'fax': '+1.2083895771', 'email': 'email@markmonitor.com',
'changedate': '2023-09-07T07:32:23.899353Z', 'country': 'US', 'registrar': 'MARKMONITOR Inc.'},
{'handle': 'GIHU101-FRNIC', 'type': 'ORGANIZATION', 'name': 'Google Ireland Holdings Unlimited Company',
'street1': "70 Sir John Rogerson's Quay", 'street2': '2 Dublin', 'street3': None, 'street4': None,
'country': 'IE', 'phone': '+353.14361000', 'fax': None, 'email': 'dns-admin@google.com',
'changedate': '2018-03-02T17:52:06Z nic.fr'},
{'handle': 'MI3669-FRNIC', 'type': 'ORGANIZATION', 'name': 'MarkMonitor Inc.',
'street1': '2150 S. Bonito Way, Suite 150', 'street2': '83642 Meridian', 'street3': 'ID',
'street4': None, 'country': 'US', 'phone': '+1.2083895740', 'fax': '+1.2083895771',
'email': 'ccops@markmonitor.com', 'changedate': '2021-10-05T15:17:57Z nic.fr'}]
'phone': '+353.14361000', 'fax': None, 'email': 'email@google.com', 'changedate': None, 'country': 'IE',
'registrar': 'MARKMONITOR Inc.'}]

assert res == expected


Expand Down Expand Up @@ -669,3 +633,20 @@ def test_domain_command(args: dict[str, Any], expected_res, mocker: MockerFixtur
reliability='B - Usually reliable'
)
assert len(result) == expected_res


def test_parse_nic_contact_new_regex():
"""
Given:
- Data fetched from the API.
When:
- calling the whois/domain command.
Then:
- validate that the data extracted without timeout.
"""
from Whois import parse_nic_contact
data = pathlib.Path('test_data/whois_response_text.txt').read_text()
res = parse_nic_contact([data])
assert len(res) == 2
assert any(entry.get('email') == 'test@test.net' for entry in res)
assert any(entry.get('country') == 'TEST' for entry in res)
108 changes: 108 additions & 0 deletions Packs/Whois/Integrations/Whois/test_data/whois_response.txt
View file
@@ -0,0 +1,108 @@
%%
%% This is the AFNIC Whois server.
%%
%% complete date format: YYYY-MM-DDThh:mm:ssZ
%%
%% Rights restricted by copyright.
%% See https://www.afnic.fr/en/domain-names-and-support/everything-there-is-to-know-about-domain-names/find-a-domain-name-or-a-holder-using-whois/
%%
%%

domain: google.fr
status: ACTIVE
eppstatus: serverUpdateProhibited
eppstatus: serverTransferProhibited
eppstatus: serverDeleteProhibited
eppstatus: serverRecoverProhibited
hold: NO
holder-c: GIHU100-FRNIC
admin-c: GIHU101-FRNIC
tech-c: MI3669-FRNIC
registrar: MARKMONITOR Inc.
Expiry Date: 2023-12-30T17:16:48Z
created: 2000-07-26T22:00:00Z
last-update: 2022-12-03T09:40:42.40624Z
source: FRNIC

nserver: ns1.google.com
nserver: ns2.google.com
nserver: ns3.google.com
nserver: ns4.google.com
source: FRNIC

registrar: MARKMONITOR Inc.
address: 2150 S. Bonito Way, Suite 150
address: ID 83642 MERIDIAN
country: US
phone: +1.2083895740
fax-no: +1.2083895771
e-mail: email@markmonitor.com
website: http://www.markmonitor.com
anonymous: No
registered: 2002-01-07T00:00:00Z
source: FRNIC

nic-hdl: GIHU100-FRNIC
type: ORGANIZATION
contact: Google Ireland Holdings Unlimited Company
address: Google Ireland Holdings Unlimited Company
address: 70 Sir John Rogerson's Quay
address: 2 Dublin
country: IE
phone: +353.14361000
e-mail: email@google.com
registrar: MARKMONITOR Inc.
changed: 2022-10-15T05:41:14.918179Z
anonymous: NO
obsoleted: NO
eppstatus: serverUpdateProhibited
eppstatus: associated
eligstatus: not identified
reachstatus: not identified
source: FRNIC

nic-hdl: MI3669-FRNIC
type: ORGANIZATION
contact: MarkMonitor Inc.
address: 2150 S. Bonito Way, Suite 150
address: 83642 Meridian
country: US
phone: +1.2083895740
fax-no: +1.2083895771
e-mail: email@markmonitor.com
registrar: MARKMONITOR Inc.
changed: 2023-09-07T07:32:23.899353Z
anonymous: NO
obsoleted: NO
eppstatus: associated
eppstatus: active
eligstatus: ok
eligsource: REGISTRAR
eligdate: 2021-10-05T00:00:00Z
reachstatus: ok
reachmedia: email
reachsource: REGISTRAR
reachdate: 2021-10-05T00:00:00Z
source: FRNIC

nic-hdl: GIHU101-FRNIC
type: ORGANIZATION
contact: Google Ireland Holdings Unlimited Company
address: 70 Sir John Rogerson's Quay
address: 2 Dublin
country: IE
phone: +353.14361000
e-mail: email@google.com
registrar: MARKMONITOR Inc.
anonymous: NO
obsoleted: NO
eppstatus: associated
eppstatus: active
eligstatus: not identified
reachstatus: ok
reachmedia: email
reachsource: REGISTRAR
reachdate: 2018-03-02T00:00:00Z
source: FRNIC

>>> WHOIS request date: 2023-09-13T09:36:56.921026Z <<<
78 changes: 78 additions & 0 deletions Packs/Whois/Integrations/Whois/test_data/whois_response_text.txt
View file
@@ -0,0 +1,78 @@
%%
%% This is the AFNIC Whois server.
%%
%% complete date format: YYYY-MM-DDThh:mm:ssZ
%%
%% Rights restricted by copyright.
%% See test
%%
%%

domain: test.fr
status: TEST
eppstatus: TEST
hold: TEST
holder-c: TEST
admin-c: TEST
tech-c: TEST
registrar: TEST
Expiry Date: 2026-06-09T14:25:21.617352Z
created: 2026-06-09T14:25:21.638904Z
last-update: 2027-06-16T13:38:17.476151Z
source: TEST

nserver: test.com
nserver: test.com
source: TEST

registrar: TEST
address: TEST 1
address: TEST
country: TEST
phone: +12.1234
fax-no: +12.1234
e-mail: test@test.net
website: http://www.test.net/tld/fr
anonymous: TEST
registered: 2006-07-24T00:00:00Z
source: TEST

nic-hdl: TEST
type: TEST
contact: TEST
address: TEST
address: 9 TEST
address: 1234 TEST
country: TEST
phone: +12.1234
e-mail: test@test.fr
registrar: TEST
anonymous: TEST
obsoleted: TEST
eppstatus: TEST
eppstatus: TEST
eligstatus: TEST
reachstatus: TEST
source: TEST

nic-hdl: TEST
type: TEST
contact: TEST
address: TEST
address: TEST 1
address: TEST
country: TEST
phone: +12.1234
e-mail: test@test.net
registrar: TEST
changed: 2023-09-09T22:59:54.000448Z
anonymous: TEST
obsoleted: TEST
eppstatus: TEST
eppstatus: TEST
eligstatus: TEST
reachstatus: TEST
source: TEST

>>> WHOIS request date: 2023-09-11T11:59:51.886322Z <<<

8 changes: 8 additions & 0 deletions Packs/Whois/ReleaseNotes/1_5_3.md
View file
@@ -0,0 +1,8 @@

#### Integrations

##### Whois
- Updated the Docker image to: *demisto/ippysocks-py3:1.0.0.74506*.

- Fixed an issue where extraction on some domains caused timeout issue.

0 comments on commit 16f1817

Please sign in to comment.