Merge branch 'master' into feat_aiohttp_update

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 2.73.0
+current_version = 2.74.0
 commit = True
 tag = True
 tag_message = 

CHANGELOG.rst

@@ -4,6 +4,42 @@
 Synapse Changelog
 *****************
 
+v2.74.0 - 2021-12-08
+====================
+
+Features and Enhancements
+-------------------------
+- Add ``.onion`` and ``.bit`` to the TLD list used for scraping text. Update
+  the TLD list from the latest IANA TLD list.
+  (`#2483 <https://github.com/vertexproject/synapse/pull/2483>`_)
+  (`#2497 <https://github.com/vertexproject/synapse/pull/2497>`_)
+- Add support for writeback mirroring of layers.
+  (`#2463 <https://github.com/vertexproject/synapse/pull/2463>`_)
+  (`#2489 <https://github.com/vertexproject/synapse/pull/2489>`_)
+- Add ``$lib.scrape()`` Stormtypes API. This can be used to do programmatic
+  scraping of text using the same regular expressions used by the Storm
+  ``scrape`` command and the ``synapse.lib.scrape`` APIs.
+  (`#2486 <https://github.com/vertexproject/synapse/pull/2486>`_)
+- Add a ``jsonlines`` output mode to Cortex streaming HTTP endpoints.
+  (`#2493 <https://github.com/vertexproject/synapse/pull/2493>`_)
+- Add a ``--raw`` argument to the Storm ``pkg.load`` command. This loads the
+  raw JSON response as a Storm package.
+  (`#2491 <https://github.com/vertexproject/synapse/pull/2491>`_)
+- Add a ``blocked`` enum to the ``proj:ticket:status`` property to represent a
+  blocked ticket.
+  (`#2490 <https://github.com/vertexproject/synapse/pull/2490>`_)
+
+Bugfixes
+--------
+- Fix a behavior with ``$path`` losing variables in pure Storm command
+  execution.
+  (`#2492 <https://github.com/vertexproject/synapse/pull/2492>`_)
+
+Improved Documentation
+----------------------
+- Update the description of the Storm ``scrape`` command.
+  (`#2494 <https://github.com/vertexproject/synapse/pull/2494>`_)
+
 
 v2.73.0 - 2021-12-02
 ====================

docs/synapse/userguides/storm_ref_cmd.rstorm

@@ -1094,37 +1094,44 @@ The ``reindex`` command reindexes a given node property. This is an administrati
 scrape
 ------
 
-The ``scrape`` command parses one or more secondary properties or variables of the inbound node(s), attempts to identify ("scrape") common forms from the content, and returns nodes for the identified forms (creating the nodes if they do not already exist). This is useful (for example) to extract forms such as email addresses, domains, URLs, hashes, etc. from unstructured text.
+The ``scrape`` command parses one or more secondary properties of the inbound node(s) and attempts to identify ("scrape") common forms from the content, creating the nodes if they do not already exist. This is useful (for example) for extracting forms such as email addresses, domains, URLs, hashes, etc. from unstructured text.
 
-``scrape`` can optionally link the source nodes(s) to the scraped forms via ``refs`` light edges.
+The ``--refs`` switch can be used to optionally link the source nodes(s) to the scraped forms via ``refs`` light edges.
 
-By default, the scrape command will yield the nodes that it received as input (i.e., the command will not consume nodes by default) to allow command chaining.  However if executed with the option ``--yield`` the ``scrape`` command will yield the **scraped nodes** rather than the input nodes.
+By default, the ``scrape`` command will return the nodes that it received as input. The ``--yield`` option can be used to return the scraped nodes rather than the input nodes.
 
 
 .. storm-cli:: scrape --help
 
 
 **Examples:**
 
-- Scrape common forms and create nodes from the body of domain WHOIS record(s) for domain woot.com:
-
+- Scrape the text of WHOIS records for the domain ``woot.com`` and create nodes for common forms found in the text:
 
 .. storm-cli:: inet:whois:rec:fqdn=woot.com | scrape :text
 
 
-- Scrape common forms from the content of a set of Internet posts from a given account, link the created nodes to the original posts, and uniq the results:
-
+- Scrape the text of the posts from the ``scumbots`` Twitter account, link the created nodes to the original posts, and return the scraped nodes:
 
-.. storm-pre:: [inet:web:post=((twitter.com,finley1589),20160918105200) :acct=(twitter.com,finley1589) :time=20160918105200 :url=http://twitter.com/finley1589/statuses/777460180225261568 :text="THOSE THAT LOVE THE USA PRAY! https://t.co/q5EPZrxGKk "]
-.. storm-cli:: inet:web:post:acct=(twitter.com,finley1589) | scrape :text --refs --yield | uniq
+.. storm-pre:: [inet:web:post=((twitter.com,scumbots),20211120125018) :acct=(twitter.com,scumbots) :time=20211120125018 :url=https://twitter.com/ScumBots/status/1446122437150531600 :text='#ZHacker13ReverseTCPShell SHA256: c92a7f657301e496610ae9ff85e01fe8e60f1179cae6e062bfcfd191a4c0e30d C2: 23[.]88[.]123[.]250:4444' ]
+.. storm-cli:: inet:web:post:acct=(twitter.com,scumbots) | scrape :text --refs --yield | uniq
 
 
 **Usage Notes:**
 
-- If no properties or variables are specified, ``scrape`` will attempt to scrape all properties by default.
-- If any scraped secondary properties are already recognizable / common properties, those properties will also be scraped (and optionally linked, if ``--refs`` is also specified). That is, scraping the ``:md5`` secondary property of a node will return a ``hash:md5`` node with that property's value.
-- ``scrape`` will only scrape node properties; it will not scrape the content of any files that may be referenced by those properties. For example, attempting to scrape the ``:file`` property of a ``media:news`` node will scrape and extract the **value** of the property (typically a SHA256 hash) if present; it will not extract indicators from the referenced file itself.
-- With respect to cyber threat data in particular, ``scrape`` cannot identify and extract "defanged" values such as "hxxp://" for URLs or "woot[.]com" for domains.
+- If no properties to scrape are specified, ``scrape`` will attempt to scrape **all** properties of the inbound nodes by default.
+- ``scrape`` will only scrape node **properties**; it will not scrape files (this includes files that may be referenced by properties, such as ``media:news:file``). In other words, ``scrape`` cannot be used to parse indicators from a file such as a PDF.
+- ``scrape`` extracts the following forms / indicators (note that this list may change as the command is updated):
+
+  - FQDNs
+  - IPv4s
+  - Servers (IPv4 / port combinations)
+  - Hashes (MD5, SHA1, SHA256)
+  - URLs
+  - Email addresses
+  - Cryptocurrency addresses
+
+- ``scrape`` is able to recognize and account for common "defanging" techniques (such as ``evildomain[.]com``, ``myemail[@]somedomain.net``, or ``hxxp://badwebsite.org/``), and will scrape "defanged" indicators by default. Use the ``--skiprefang`` switch to ignore defanged indicators.
 
 .. _storm-service:
 

setup.py

@@ -6,7 +6,7 @@
 from setuptools import setup, find_packages
 from setuptools.command.install import install
 
-VERSION = '2.73.0'
+VERSION = '2.74.0'
 
 class VerifyVersionCommand(install):
     """Custom command to verify that the git tag matches our version"""

synapse/lib/version.py

@@ -223,6 +223,6 @@ def reqVersion(valu, reqver,
 ##############################################################################
 # The following are touched during the release process by bumpversion.
 # Do not modify these directly.
-version = (2, 73, 0)
+version = (2, 74, 0)
 verstring = '.'.join([str(x) for x in version])
 commit = ''