Skip to content

Commit

Permalink
feature: update readme
Browse files Browse the repository at this point in the history
  • Loading branch information
@jerlendds
jerlendds committed Jul 15, 2023
1 parent ec5a1e1 commit cd038d0
Show file tree
Hide file tree
Showing 3 changed files with 122 additions and 129 deletions.
79 changes: 39 additions & 40 deletions pyproject.toml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,59 +5,58 @@ build-backend = "flit.buildapi"
[project]
name = "osintbuddy"
authors = [
{name = "jerlendds", email = "jerlendsdev@proton.me"},
{name = "jerlendds", email = "theosintbuddyproject@openinfolabs.com"},
]
description = "OSINTBuddy - mine, merge, and map data for novel insights"
readme = "README.md"
classifiers = [
"Development Status :: 3 - Alpha",
"Intended Audience :: Developers",
"License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
"Programming Language :: Python :: 3 :: Only",
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10"
"License :: OSI Approved :: GNU Affero General Public License v3 (AGPLv3)",
"Programming Language :: Python :: 3.11"
]
requires-python = ">=3.7"
dynamic = ["version"]
dependencies = [
"selenium==4.9.0",
"pydantic==1.10.7",
"httpx==0.24.1"
"selenium>=4.9.0",
"pydantic>=1.10.8",
"httpx>=0.24.1",
"SQLAlchemy>=2.0.12",
"gremlinpy @ git+https://github.com/jerlendds/gremlinpy.git@7d3033e6a55ed9cb1f982ec3b58ca233e01c58e3"
]
[project.optional-dependencies]
test = [
"astroid==2.15.4",
"colorama==0.4.6",
"dill==0.3.6",
"eradicate==2.2.0",
"exceptiongroup==1.1.1",
"iniconfig==2.0.0",
"isort==5.12.0",
"lazy-object-proxy==1.9.0",
"mando==0.7.1",
"mccabe==0.7.0",
"mypy==1.3.0",
"mypy-extensions==1.0.0",
"packaging==23.1",
"platformdirs==3.5.1",
"pluggy==1.0.0",
"pycodestyle==2.10.0",
"pydocstyle==6.3.0",
"pyflakes==3.0.1",
"pylama==8.4.1",
"pylint==2.17.4",
"pytest==7.3.1",
"radon==6.0.1",
"six==1.16.0",
"snowballstemmer==2.2.0",
"toml==0.10.2",
"tomli==2.0.1",
"tomlkit==0.11.8",
"typing-extensions==4.5.0",
"vulture==2.7",
"wrapt==1.15.0"
"astroid==2.15.4",
"colorama==0.4.6",
"dill==0.3.6",
"eradicate==2.2.0",
"exceptiongroup==1.1.1",
"iniconfig==2.0.0",
"isort==5.12.0",
"lazy-object-proxy==1.9.0",
"mando==0.7.1",
"mccabe==0.7.0",
"mypy==1.3.0",
"mypy-extensions==1.0.0",
"packaging==23.1",
"platformdirs==3.5.1",
"pluggy==1.0.0",
"pycodestyle==2.10.0",
"pydocstyle==6.3.0",
"pyflakes==3.0.1",
"pylama==8.4.1",
"pylint==2.17.4",
"pytest==7.3.1",
"radon==6.0.1",
"six==1.16.0",
"snowballstemmer==2.2.0",
"toml==0.10.2",
"tomli==2.0.1",
"tomlkit==0.11.8",
"typing-extensions==4.5.0",
"vulture==2.7",
"wrapt==1.15.0",
"gremlinpy@git+https://github.com/jerlendds/gremlinpy.git@7d3033e6a55ed9cb1f982ec3b58ca233e01c58e3"
]

[project.urls]
Expand Down
38 changes: 6 additions & 32 deletions requirements.txt
View file
Original file line number Diff line number Diff line change
@@ -1,32 +1,6 @@
astroid==2.15.4
colorama==0.4.6
dill==0.3.6
eradicate==2.2.0
exceptiongroup==1.1.1
iniconfig==2.0.0
isort==5.12.0
lazy-object-proxy==1.9.0
mando==0.7.1
mccabe==0.7.0
mypy==1.3.0
mypy-extensions==1.0.0
packaging==23.1
platformdirs==3.5.1
pluggy==1.0.0
pycodestyle==2.10.0
pydocstyle==6.3.0
pyflakes==3.0.1
pylama==8.4.1
pylint==2.17.4
pytest==7.3.1
radon==6.0.1
six==1.16.0
snowballstemmer==2.2.0
toml==0.10.2
tomli==2.0.1
tomlkit==0.11.8
typing-extensions==4.5.0
vulture==2.7
wrapt==1.15.0
pydantic==1.10.7
selenium==4.9.0

selenium>=4.9.0
pydantic>=1.10.8
httpx>=0.24.1
SQLAlchemy>=2.0.12
gremlinpy @ git+https://github.com/jerlendds/gremlinpy.git@7d3033e6a55ed9cb1f982ec3b58ca233e01c58e3
134 changes: 77 additions & 57 deletions src/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,74 +2,94 @@

The plugins library for [jerlendds/osintbuddy](https://github.com/jerlendds/osintbuddy). Currently in beta, expect changes to come...

***Please note:** [OSINTBuddy plugins](https://github.com/jerlendds/osintbuddy-plugins) are still in early alpha and breaking changes may occasionally occur in the API. That said, if you remain on the `main` branch and avoid accessing protected methods we will try our best to avoid introducing breaking changes.*

### **NOTICE:** Major update to plugins, any created plugins will have to be updated to use a more convenient data access method:
- Remove `name` from `ob.Plugin`
- @todo decide on pydantic or a plain dict for passing node values by label
### **NOTICE:** There has been a major update to plugins, any created plugins will have to be updated to use the new, and more convenient data access method:
- Remove `name` from any `ob.Plugin`
- Update `node['data']` access to be `node.label_defined_in_node`
- Please see the introduction to the plugin system below

## Creating your first plugin

In this guide, we will dive into creating a custom plugin for OSINTBuddy that extends its capabilities. This will enable you to incorporate additional sources and transformations into the OSINTBuddy toolbox. We will use the provided code example as a reference to build our plugin.

### Step-by-step guide to create a new plugin

Start by importing the necessary modules:
The osintbuddy plugin system at its core is very simple. An `OBRegistry` class holds all registered `OBPlugin` classes within the application. This registry is loaded into the [osintbuddy application](https://github.com/jerlendds/osintbuddy/) where it is then used to load the available entities for the user when they access a project graph, load transforms when a user opens the context menu of a node, and perform transformations which expect a `Plugin.blueprint()` to be returned. The returned data of a transform decorated method will be automatically mapped to a [JanusGraph](https://janusgraph.org/) database through [gremlin](https://tinkerpop.apache.org/) according to the labels *(as snakecase)* you previously set in the classes `node` for whatever `Plugin.blueprint()`
you return.

To make this a bit more clear please see the below example...

```py
import socket
import osintbuddy as ob
```

Define a class for your plugin, inheriting from `ob.Plugin`. Set the required attributes such as `label`, and optionally set an `icon` like `world-www` (using [tabler-icon](https://tabler-icons.io/) names), and a `node` which contains a list of elements used as a blueprint for creating the node displayed on the OSINTBuddy UI.

```py
class WebsitePlugin(ob.Plugin):
label = 'Website'
name = 'Website'
color = '#1D1DB8'
icon = 'world-www'
from pydantic import BaseModel
import osintbuddy import transform, Plugin
from osintbuddy.elements import TextInput, DropdownInput, Title, CopyText
from osintbuddy.errors import OBPluginError


class CSESearchResults(Plugin):
label = "CSE Result"
name = "CSE result"
show_label = False # do not show this on the entities dialog
# the user sees on the left of the project graph screen
color = "#058F63"
node = [
ob.elements.TextInput(label='Domain', icon='world-www'),
Title(label="Result"),
CopyText(label="URL"),
CopyText(label="Cache URL"),
]
```

Now, define a transformation method to gather and transform the data from the input node. In this case, the transform_to_ip function uses the `socket.gethostbyname()` Python module to obtain the IP address for a given domain. Decorate the method with `@transform()` and set the metadata attributes such as `label` and `icon`.

```py
@transform(label='To IP', icon='building-broadcast-tower')
def transform_to_ip(self, node, **kwargs):
blueprint = IPAddressPlugin.blueprint(
ip_address=socket.gethostbyname(node['data'][0])
)
return blueprint
```
## Using the new plugin

To use the new plugin, simply add it to the existing OSINTBuddy plugins folder inside the application, in development mode it will be detected automatically and loaded by the platform. Once added, you can access your 'Website' plugin and use the 'To IP' transformation.

Creating a custom plugin for OSINTBuddy is an easy and effective way to enhance the capabilities of the tool, allowing you to fetch information from additional sources or transform the data in new ways. By following this guide and using the provided code example as a reference, you'll be able to create your own unique plugins to extend the functionality of OSINTBuddy to fit your specific needs.


### Full example

```py
import socket
import osintbuddy as ob

class WebsitePlugin(ob.Plugin):
label = 'Website'
name = 'Website'
color = '#1D1DB8'
icon = 'world-www'
class CSESearchPlugin(Plugin):
label = "CSE Search"
name = "CSE search"
color = "#2C7237"
node = [
ob.elements.TextInput(label='Domain', icon='world-www'),
[
TextInput(label="Query", icon="search"),
TextInput(label="Pages", icon="123", default="1"),
],
DropdownInput(label="CSE Categories", options=cse_link_options)
]

@transform(label='To IP', icon='building-broadcast-tower')
def transform_to_ip(self, node, **kwargs):
blueprint = IPAddressPlugin.blueprint(
ip_address=socket.gethostbyname(node['data'][0])
)
return blueprint
@transform(label="To cse results", icon="search")
async def transform_to_cse_results(
self,
node: BaseModel, # dynamically generated pydantic model
# that is mapped from the above labels contained within `node`
use # a pydantic model allowing you to access a selenium instance
# (and eventually a gremlin graph and settings api)
):
results = []

if not node.query:
raise OBPluginError((
'You can send error messages to the user here'
'if they forget to submit data or if some other error occurs'
))

# notice how you can access data returned from the context menu
# of this node; using the label name in snake case
print(node.cse_categories, node.query, node.pages)

... # (removed code for clarity)

if resp:
for result in resp["results"]:
url = result.get("breadcrumbUrl", {})
# some elements you can store more than just a string,
# (these elements storing dicts are mapped
# to janusgraph as properties with the names
# result_title, result_subtitle, and result_text)
blueprint = CSESearchResults.blueprint(
result={
"title": result.get("titleNoFormatting"),
"subtitle": url.get("host") + url.get("crumbs"),
"text": result.get("contentNoFormatting"),
},
url=result.get("unescapedUrl"),
cache_url=result.get("cacheUrl"),
)
results.append(blueprint)
# here we return a list of blueprints (blueprints are dicts)
# but you can also return a single blueprint without a list
return results

```


0 comments on commit cd038d0

Please sign in to comment.