add support for analysis of source code/scripted languages

[adamstorek]

This enhancement extends capa's functionality to the analysis of potentially malicious scripts and source code. A tree-sitter backend was added to parse the source files into a lightweight AST. Features akin to the PE-Vivisect capa are then extracted:

File-level:

• trivial: language, file format
• global string literals
• global integer literals
• namespaces
• globally-instantiated imported classes
• globally-called imported functions

Function-level:

• string literals
• integer literals
• imported classes
• imported functions

To install Tree-sitter:

1. Pip-install Tree-sitter:
   pip3 install tree-sitter
2. Install bindings:
   mkdir vendor build
cd vendor
git clone git@github.com:tree-sitter/tree-sitter-c-sharp.git
git clone git@github.com:tree-sitter/tree-sitter-embedded-template.git
git clone git@github.com:tree-sitter/tree-sitter-html.git
git clone git@github.com:tree-sitter/tree-sitter-javascript.git

Checklist

• No CHANGELOG update needed

• No new tests needed

• No documentation update needed

[github-actions]

Please add bug fixes, new features, breaking changes and anything else you think is worthwhile mentioning to the master (unreleased) section of CHANGELOG.md. If no CHANGELOG update is needed add the following to the PR description: [x] No CHANGELOG update needed

[williballenthin]

This is a great start towards adding scripting language support to capa! thanks @adamstorek!

The code is already quite good and I don't anticipate any major issues to getting it merged; however, I have added a number of comments on regions I think should be tweaked.

One thing you should know about me is that I prefer to over-communicate review feedback with the understanding that everything is up for discussion. So, if anything feels weird or wrong, don't hesitate to ask for more details or deeper discussion.

General points:

1. I really like the file range address type, I think that will work well.
2. I think we can simplify the code a bit by merging the "Script" feature into "Format". thoughts?
3. Some of the embedded data and configuration can be restructured into python globals.
4. I'd like to hear a bit about what it takes to embed/depend on the TS languages so we can ensure its easy for people to download/use.
5. Please add tests showing the features extracted on various example files

[williballenthin]

could we use format for this? e.g. format: C#.

pro:

• fewer features to memorize
• less duplication
• less code

con:

• maybe slightly less precise

[adamstorek]

The problem with overloading the file format feature is that file to language is a one-to-many mapping, e.g. there can be embedded templates that contain multiple different script languages such as C# for server-side scripts and JavaScript for client-side.

[williballenthin]

we should also think about maybe trying to guess the language based on the file contents if there's no extension.

also supporting things like .cs_ which some may use to prevent the file from accidentally getting executed.

[adamstorek]

Auto-identifying the script language is definitely a worthwhile feature which I have deprioritized for the minimal implementation and instead manually incorporated the extensions (now including the defanged extensions as suggested above). This is also because it might not always be very straightforward to do so (e.g. one file might include multiple scripts; sometimes context might be necessary).

[williballenthin]

i think it would be worthwhile to get the tests running (and passing) in CI. this means:

• add the example files to capa-testfiles and get those merged, and
• update the github actions workflows to install the TS bindings (temporarily, until we have a better solution)

[adamstorek]

• add the example files to capa-testfiles and get those merged, and

Just submitted the pull request pull request.

• update the github actions workflows to install the TS bindings (temporarily, until we have a better solution)

On it.

[mike-hunhoff]

Can we document why we are only/specially handling ASPX here?

[adamstorek]

This is related to the issue discussed here: #1080 (comment).

[mike-hunhoff]

nice updates! I've left a few comments, questions, and suggestions for your review 🚀

[mike-hunhoff]

Can we add a comment on what are assumptions are here? I'm not overly familiar with tree-sitter but it appears that we assume it will only throw errors when encountering a language mismatch e.g. we attempt to parse Python using tree-sitter C# tooling?

[adamstorek]

This might be more readable, what do you think?

def _parse(ts_language: Language, buf: bytes) -> Optional[Tree]:
    try:
        parser = Parser()
        parser.set_language(ts_language)
        return parser.parse(buf)
    except ValueError:
        return None


def _contains_errors(ts_language, node: Node) -> bool:
    return ts_language.query("(ERROR) @error").captures(node)


def get_language_ts(buf: bytes) -> str:
    for language, ts_language in TS_LANGUAGES.items():
        tree = _parse(ts_language, buf)
        if tree and not _contains_errors(ts_language, tree.root_node):
            return language
    raise ValueError("failed to parse the language")

[mike-hunhoff]

does this mean we only support Linux?

[adamstorek]

Tree-sitter needs to compile its (C) language bindings. Although I have a limited knowledge of package management, I've suggested to Moritz that we should precompile and package the supported tree-sitter bindings for each platform we support. The current state is a temporary measure.

[mike-hunhoff]

can this call generate any exceptions?

[adamstorek]

It can throw type errors (which I believe we prevent with mypy) and a value error if parsing completely fails. Then the parse method will throw a ValueError, so the engine will throw a ValueError etc.: I can handle it in the following way at the Extractor level:

try:
    self.language = capa.features.extractors.ts.autodetect.get_language(path)
    self.template_engine = self.get_template_engine(buf)
    self.engines = self.get_engines(buf)
except ValueError as e:
    raise UnsupportedFormatError(e)

[mike-hunhoff]

The intended use of this function appears to be decoding a string found in a specific byte range? If so, consider changing the name to something more descriptive like get_str_from_range. Also, do we expect encoding exceptions to be thrown by the decode?

[adamstorek]

Good question, but I doubt tree-sitter would be able to parse something that we can't decode.

[adamstorek]

Also changed get_range to get_str.

[mike-hunhoff]

need more discussion on #1125

[mike-hunhoff]

typo w/ _?

[adamstorek]

No, this is merely a private method to handle import table lookups that the public is_import method uses.

[mike-hunhoff]

do we expect exceptions to occur here?

[adamstorek]

This can raise ValueError (and does when TS labels something as an int which is not an int), but the ValueError is handled by the caller (see extract_integers).