Interactive Interface for exploring threat intelligence

[18z]

Hi guys,

When investigating the intelligence, our steps are often the iteration of extracting info and then analyzing the info.

For instance, when digging into a binary, we might want to know if there are base64 encoded strings inside. So our first step would be extract encoded strings that are most likely base64 encoded. Then we can start to analyze the decoded strings.

Or it could be the scenario that we'd like to find (extract) URLs inside the binary and then analyze them.

I think IntelOwl is doing very well of what mentioned above. And I think it would be so nice if we can explore the threat intelligence with enhenced pyintelowl.

To do so, we need to design more Intelowl APIs making the data extraction and piping it to the analyzers more easily.

Here's the concept.

$> open /home/test/binaryfile
file successfully opened
session id: 1345873

$> file.Type()
binaryfile: Dalvik dex file version 035

$> URLs = file.findURLs()
["https://a.b.c", "https://a.a.c"]

$> InspectURL(URLs[1])
Then show analysis results from URL related analyzers. 

To make the above concept works, we need to

1. Design APIs that can extract basic information from the tartget. (Binary is just one of a kind)
2. Design APIs that make the analyzers easy to be used. Easy to pipe in the data and easy to read and reuse the results.

So what do you guys think? :D
Please correct me if I'm wrong with something.

[mlodic]

Hey,

I really appreciate your suggestion.

Right now IntelOwl does not provide a real threat intelligence platform. It just allows to retrieve data from different sources/tools at the same time. The main goal is and was to just to speed up data retrieval. So IntelOwl is agnostic in respect of each source/tool. It just gets the data and ignores completely how each tool works and which is the goal of each tool.

Recently, with the increase of the success of the project, we started to think on how we could transform IntelOwl in a real threat intelligence platform that can be used for complex analysis. To do that we would need to add an important feature:

• provide the chance to pivot through IoCs.

An IntelOwl user should be able to jump from an IOC extracted from a specific analysis to another one from a subsequent analysis....and so on until the analyst thinks that he got all the needed data and the "investigation" is finished. IntelOwl should also provide the chance to build and customize such flows and these flows should be executed automatically, re-used and shared. There is a first concept here.

This feature would still leave to the user the duty to understand which analyzers he is more interested of, which data to extract, which data better correlate with others and which flows to create.
In the case that you showed me, you wanted to build a very specific flow to solve a problem that comes from your experience and from your work field. Such kind of flows should be sharable, like I mentioned, cause there are a lot of other people in the world with the same issues, but should also be customizable.

What I mean is: if IntelOwl provided the chance to create such kind of investigations easily, it should not be biased about the extracted data. In other words, IntelOwl should not tell to the user that he would need that specific data cause only the user knows which data he needs. IntelOwl should just provide the chance to build complex flows and investigations, like the one that you mentioned. We should focus on this because IntelOwl supports too many different analyzers and tools and too many different use cases that would be really unmanagable to try to address all of them.

Then, I agree that there are some specific flows that are more popular than others (like the one you suggested) and, in some way, we should help these cases more than the others.

Another point of view that I have is that I don't see pyintelowl like something that could become a real complex CLI with a lot of features cause my vision is to mainly provide a strong GUI that is accessible to more types of users, with different expertise.
I see pyintelowl mainly as a library to integrate IntelOwl with other tools. The actual existing CLI is just an helper tool and, with the efforts we are making for the new upcoming v4 release with a complete new interface, we should bring the user attention to use the GUI and to avoid the CLI cause the experience would be really better in the GUI.

So, what do you think about having the chance to build flows like you mentioned from the GUI so they can be repeatable and re-usable and to leverage only the analyzers that you think are worth for your goal. That would be a custom "investigation" of yours that you could keep for yourself or, maybe, you could also share in a public repository of "investigations" types so everyone can use it.
Then, you could also replicate that investigation via API programmatically. But that would be just a call like that:

IntelOwl.execute_investigation("<name_of_the_custom_investigation_you_created")

and you would get, as a result, only the data from the executed analyzers that you think are worth (these would be specified in the "investigation" declaration).

Obviously, it is not so easy to design such kind of customizable "investigation" feature and we don't have anything as a draft yet. But we aim to do something like that in the future, probably next year.

Did I get your point or I wandered? :P Makes sense?

I'll also move this issue to a discussion

[18z]

Hey bro,

Thanks for spending the effort explaining the vision of IntelOwl and how you guys thought about it.

I'm happy to see that we mutual goal which is to provide the chance to pivot through IoCs. And I agree with you to bring the attention of the users to the GUI for better experiences.

So here I made a scatch of visualizing and building the "flows".

In the skatch above, A1 to A7 are the analyzers. They can be dragged and dropped to build the flow. R1 to R4 are the outputs from the analyzers. The black arrow line pipes the data to the analyzers. And the blue arrow lines pipe out the output from the analyzers.

For example:

• First, we upload the initial data, and to the best of users' experience, they choose A4 analyzer to analyze the data. So they drag block A4 from the above panel and drop it anywhere in the empty panel. Then, users linked an arrow line from initial data to A4. This means users make initial data as the input resource for analyzer A4. After the analysis, A4 analyzer outputs 3 results (R1, R2 and R3).
• Then, users think R1 is worth to dig deeper and is suitable using A7 analyzer for digging. So they do the drag, drop and link again for the digging and the result is R4. Then so on and so forth till the users think they get what they want.

So, by dragging, dropping the analyzers and linking from data to the analyzers. This builds up the flow.

Once the user finish their "investigation". We can save the "flow" with json format playbook. Further, with the visualized "flow", it helps to clarify and sort up the "clues". Last but not least, with this idea, the "flow" can be build, re-use, share and READ.

:D

[mlodic]

I really love your help. I think that the concept you proposed is the direction we should take. That would require serious and complex design, both frontend and backend side but definitively this is something that would make completely sense and would make IntelOwl a real investigative tool.

Right now, we are planning to conclude and make a major release with the rewritten GUI and some other features...everything should be completed at the end of the GSoC period.

I would say that next year could be the time to tackle this, starting with some design at the end of the year. I'll update this thread with the progress

PS:
In some way, this would be pretty similar to what does Tines which is also a sponsor of IntelOwl.

[18z]

That's great! I'm happy that you like this idea and I want this to come true so badly.
Please keep me updated and please let me know if there's anything I can help. 😄