The :pyFoliumMap<msticpy.vis.foliummap.FoliumMap> class is a wrapper around the Folium geo-mapping interactive mapping package. See Folium.
The MSTICPy Folium plotting can be used with DataFrames, IP addresses, locations, and geohashes as well as the MSTICPy IpAddress and GeoLocation entities.
You can use Folium via a pandas accessor, the plot_map function or directly interacting with our FoliumMap class.
MSTICPy uses pandas accessors to expose a lot of its visualization functions.
Plotting with Folium can be done directly from a pandas DataFrame using the mp_plot.folium_map extension.
See :pyplot_map <msticpy.vis.folium_map.plot_map>
This function returns an instance of the :pyFoliumMap<msticpy.vis.foliummap.FoliumMap> class that you can further customize (see FoliumMap class later in the document)
We'll read in some IpAddress data to use in the examples
# read in a DataFrame from a csv file
geo_loc_df = (
pd
.read_csv("data/ip_locs.csv", index_col=0)
.dropna(subset=["Latitude", "Longitude"]) # We need to remove any NaN values
)
display(geo_loc_df.head(5))| CountryCode | CountryName | State | City | Longitude | Latitude | Asn | IpAddress |
|---|---|---|---|---|---|---|---|
| US | United States | Virginia | Boydton |
|
|
|
65.55.44.109 |
| CA | Canada | Ontario | Toronto |
|
|
|
13.71.172.128 |
| CA | Canada | Ontario | Toronto |
|
|
|
13.71.172.130 |
| US | United States | Texas | San Antonio |
|
|
|
40.124.45.19 |
| US | United States | Iowa | Des Moines |
|
|
|
104.43.212.12 |
geo_loc_df.mp_plot.folium_map(ip_column="IPAddress")
If you already have coordinates in the data you can use these (rather than looking up the IP location again) using the lat_column and long_column parameters.
geo_loc_df.mp_plot.folium_map(
lat_column="Latitude", long_column="Longitude", zoom_start=10
)
You can use the Folium layers feature, specifying a column value on which to group each layer with the layer_column parameter.
geo_loc_df.mp_plot.folium_map(
ip_column="IPAddress", layer_column="CountryName", zoom_start=2
)
You use DataFrame column values to populate the tooltip and popup elements for each marker with the tooltip_columns and popup_columns parameters.
Although these parameters are shown being used together you can either of them separately.
First we add some random data to our DataFrame
# Create some data to display
data_df = pd.DataFrame({
"Status": ["Home", "Office", "Vacation"] * (len(geo_loc_df) // 3),
"Friendliness": ["Warm", "Cold", "Medium"] * (len(geo_loc_df) // 3),
"Flavor": ["Chocolate", "Cinnamon", "Mango"] * (len(geo_loc_df) // 3),
"SpiceLevel": [1, 2, 3] * (len(geo_loc_df) // 3)
})
geo_loc_data_df = pd.concat([geo_loc_df, data_df], axis=1)Then plot using the tooltip_columns and popup_columns parameters
geo_loc_data_df.mp_plot.folium_map(
ip_column="IPAddress",
layer_column="CountryName",
tooltip_columns=["Status", "Flavor"],
popup_columns=["Friendliness", "SpiceLevel", "Status", "Flavor"],
zoom_start=2,
)
You can also control the icons used for each marker with the icon_column parameters. If you happen to have a column in your data that contains names of FontAwesome or GlyphIcons icons. More typically you would combine the icon_column with the icon_map parameter. You can specify either a dictionary or a function. For a dictionary, the value of the row in icon_column is used as a key - the value is a dictionary of icon parameters passed to the Folium.Icon class. For a method, the icon_column value is passed as a single parameter and the return value should be a dictionary of valid parameters for the Icon class. You can read the documentation for this function in the doc.
If icon_map is a dict it should contain keys that map to the value of icon_col. The values of te dict should be dicts of valid folium Icon properties ("color", "icon_color", "icon", "angle", "prefix"). The dict should include a "default" entry that will be used if the value in the DataFrame[icon_col] doesn't match any key. For example:
icon_map = {
"high": {
"color": "red",
"icon": "warning",
},
"medium": {
"color": "orange",
"icon": "triangle-exclamation",
"prefix": "fa",
},
"default": {
"color": "blue",
"icon": "info-sign",
},
}If icon_map is a function it should take a single str parameter (the item key) and return a dict of icon properties. It should return a default set of values if the key does not match a known key. The icon_col value for each row will be passed to this function and the return value used to populate the Icon arguments.
For example:
def icon_mapper(icon_key):
if icon_key.startswith("bad"):
return {
"color": "red",
"icon": "triangle-alert",
}
...
else:
return {
"color": "blue",
"icon": "info-sign",
}Check out the possible names for icons:
- FontAwesome icon (prefix "fa") names are available at https://fontawesome.com/
- GlyphIcons icons (prefix "glyphicon") are available at https://www.glyphicons.com/
icon_map = {
"US": {
"color": "green",
"icon": "flash",
},
"GB": {
"color": "purple",
"icon": "flash",
},
"default": {
"color": "blue",
"icon": "info-sign",
},
}
geo_loc_df.mp_plot.folium_map(
ip_column="IPAddress",
icon_column="CountryCode",
icon_map=icon_map,
zoom_start=2,
)
The plot_map function is identical to the mp_plot.folium_map accessor. You can import this directly and use in place of the pandas accessor.
from msticpy.vis.foliummap import plot_map
plot_map(
data=geo_loc_df,
ip_column="IPAddress",
icon_column="CountryCode",
icon_map=icon_map,
zoom_start=2,
)Use the :pyFoliumMap class <msticpy.vis.foliummap.FoliumMap> when you want to build up data clusters and layers incrementally.
It now supports multiple data types for entry:
- IpAddress entities (:py
add_ip_cluster <msticpy.vis.foliummap.FoliumMap.add_ip_cluster>) - Geolocation entities (:py
add_ip_cluster <msticpy.vis.foliummap.FoliumMap.add_geoloc_cluster>) - IP addresses (:py
add_ip_cluster <msticpy.vis.foliummap.FoliumMap.add_ips>) - Locations (:py
add_ip_cluster <msticpy.vis.foliummap.FoliumMap.add_locations>) - GeoHashes (:py
add_ip_cluster <msticpy.vis.foliummap.FoliumMap.add_geo_hashes>)
You can also use other member functions to add layers and cluster groups see :pyFoliumMap class <msticpy.vis.foliummap.FoliumMap> for more details.
Here is the documentation for initializing the class.
FoliumMap(
title: str = 'layer1',
zoom_start: float = 2.5,
tiles=None,
width: str = '100%',
height: str = '100%',
location: list = None,
)
Wrapper class for Folium/Leaflet mapping.
Parameters
----------
title : str, optional
Name of the layer (the default is 'layer1')
zoom_start : int, optional
The zoom level of the map (the default is 7)
tiles : [type], optional
Custom set of tiles or tile URL (the default is None)
width : str, optional
Map display width (the default is '100%')
height : str, optional
Map display height (the default is '100%')
location : list, optional
Location to center map on
Attributes
----------
folium_map : folium.Mapfolium_map = FoliumMap(location=(47.5982328,-122.331), zoom_start=14)
folium_mapThe underlying folium map object is accessible as the folium_map attribute
type(folium_map.folium_map)msticpy.vis.foliummap.FoliumMap :
fol_map.add_ip_cluster(
ip_entities: Iterable[msticpy.datamodel.entities.IpAddress],
**kwargs,
)import pickle
with open(b"data/ip_entities.pkl", "rb") as fh:
ip_entities = pickle.load(fh)
ip_entities = [ip for ip in ip_entities if ip.Location and ip.Location.Latitude]
folium_map = FoliumMap(zoom_start=9)
folium_map.add_ip_cluster(ip_entities=ip_entities, color='orange')
folium_map.center_map()
folium_map
Signature
ip_map.add_ips(ip_addresses: Iterable[str], **kwargs)Example
ips = geo_loc_df.query("State == 'California'").IPAddress.values
print("IP dataset", ips[:3], "...")
ip_map = FoliumMap(zoom_start="3")
ip_map.add_ips(ips)
ip_map.center_map()
ip_mapSignature
ip_map.add_locations(locations: Iterable[Tuple[float, float]], **kwargs)Example
locations = geo_loc_df.query("CountryCode != 'US'").apply(lambda x: (x.Latitude, x.Longitude), axis=1).values
print("Location dataset", locations[:3], "...")
ip_map.add_locations(locations)
ip_map.center_map()
ip_map# Create IP and GeoLocation Entities from the dataframe
def create_ip_entity(row):
ip_ent = IpAddress(Address=row["IPAddress"])
geo_loc = create_geo_entity(row)
ip_ent.Location = geo_loc
return ip_ent
def create_geo_entity(row):
# get subset of fields for GeoLocation
loc_props = row[["CountryCode", "CountryName","State", "City", "Longitude", "Latitude"]]
return GeoLocation(**loc_props.to_dict())
geo_locs = list(geo_loc_df.apply(create_geo_entity, axis=1).values)
ip_ents = list(geo_loc_df.apply(create_ip_entity, axis=1).values)
fmap_ips = FoliumMap()
fmap_ips.add_ip_cluster(ip_entities=ip_ents[:20], color='blue')
fmap_ips.center_map()
fmap_ipsfmap_ips.add_ip_cluster(ip_entities=ip_ents[30:40], color='red', icon="flash")
fmap_ips.center_map()
fmap_ipsBy default folium uses the information icon (i). Icons can be taken from the default Bootstrap set. See the default list here glyphicons
Alternatively you can use icons from the Font Awesome collection by adding prefx="fa” and icon="icon_name” to the call to add_ip_cluster or add_geo_cluster.
fmap_ips.add_geoloc_cluster(
geo_locations=geo_locs[40:50],
color='darkblue',
icon="desktop",
prefix="fa"
)
fmap_ips.center_map()
fmap_ips
from msticpy.vis.foliummap import get_map_center, get_center_ip_entities, get_center_geo_locs
print(get_center_geo_locs(geo_locs))
print(get_center_geo_locs(geo_locs, mode="mean"))
# get_map_center Will accept iterable of any entity type that is either
# an IpAddress entity or an entity that has properties of type IpAddress
print(get_map_center(ip_ents[30:40]))
print(get_map_center(ip_ents[:20]))
print(get_center_ip_entities(ip_ents[:20]))(38.7095, -79.4195) (40.01865529411764, -84.9259) (38.7095, -78.1539) (38.7095, -86.5161) (38.7095, -86.5161)
from msticpy.context.geoip import entity_distance
print("Distance between")
print(f"{ip_ents[0].Address} ({ip_ents[0].Location.City})")
print(f"{ip_ents[1].Address} ({ip_ents[1].Location.City})")
print(entity_distance(ip_ents[0], ip_ents[1]), "km", "\n")
print("Distance between")
print(f"{ip_ents[0].Address} ({ip_ents[0].Location.City})")
print(f"{ip_ents[13].Address} ({ip_ents[13].Location.City})")
print(entity_distance(ip_ents[0], ip_ents[13]), "km", "\n")Distance between 65.55.44.109 (Boydton) 13.71.172.128 (Toronto) 784.604908273247 km
Distance between 65.55.44.109 (Boydton) 131.107.147.209 (Redmond) 3751.614749340525 km
This example uses an function to create :pyIpAddress<msticpy.vis.entityschema.IpAddress> entities from IP string or IpAdddress objects stored in a pandas DataFrame.
The format_ip_entity function converts the IP string value to an IpAddress object, looks up the geo-location (this data is populated into the IpAddress Location attribute). It also adds additional information from the pandas DataFrame into the AdditionalData attribute. Values populated here will be displayed in the tooltip for each plotted IP Address on the map.
Note
The values populated into the AdditionalData attribute are specific to this example. AdditionalData is a dictionary object so you can put data keyed with a string name here. Primitive value types such as string, numbers, dates, etc. should display correctly.
# Create a IP Geolookup class
iplocation = GeoLiteLookup()
def format_ip_entity(row, ip_col):
ip_entity = entities.IpAddress(Address=row[ip_col])
iplocation.lookup_ip(ip_entity=ip_entity)
ip_entity.AdditionalData["protocol"] = row.L7Protocol
if "severity" in row:
ip_entity.AdditionalData["threat severity"] = row["severity"]
if "Details" in row:
ip_entity.AdditionalData["threat details"] = row["Details"]
return ip_entity
ips_out = list(selected_out.apply(lambda x: format_ip_entity(x, "dest"), axis=1))
ips_in = list(selected_in.apply(lambda x: format_ip_entity(x, "source"), axis=1))
ips_threats = list(ti_ip_results.apply(lambda x: format_ip_entity(x, "Ioc"), axis=1))
icon_props = {"color": "green"}
for ips in host_entity.public_ips:
ips.AdditionalData["host"] = host_entity.HostName
folium_map.add_ip_cluster(ip_entities=host_entity.public_ips, **icon_props)
icon_props = {"color": "blue"}
folium_map.add_ip_cluster(ip_entities=ips_out, **icon_props)
icon_props = {"color": "purple"}
folium_map.add_ip_cluster(ip_entities=ips_in, **icon_props)
icon_props = {"color": "red"}
folium_map.add_ip_cluster(ip_entities=ips_threats, **icon_props)
display(folium_map)../data_acquisition/GeoIPLookups