🐞 Overview

Produce a proof-of-concept for viewing Pepys data in a Jupyter notebook.

This will de-risk the future use of Jupyter notebooks both in Pepys and in general usage by analysts, offering lessons learned in data connectivity, data processing, and visualisation.

Time-permitting, to include:

• connecting to Pepys (Postgres)
• extracting State data for a period of time from one or more platforms
• giving initial visualisations of
  • spatial perspective (tracks)
  • temporal perspective (speed vs time & range vs time)
• data-pipeline to perform some cleaning of data (smooth speed?)
• perform some SciKit-Learn analysis on data to produce some new calculated variable (TBD, see below)
• push some calculated data back into Pepys
• any thoughts on offline mapping (more below)

🔗 Feature

This represents an alternate solution for #859

🔢 Acceptance criteria

Machine Learning

SciKit provides capable clustering algorithms. But, we need to think of an application of this method to Pepys data

• We could try to apply clustering to course and speed data (or change of course/speed) to identify "hi-tempo" periods for a platform, and shade the track accordingly.
• Use frequency of comments to infer tempo of operations (note: I don't see this as M/L).
• Some sentiment analysis/clustering of comments?

Offline mapping

Pepys will frequently be used without an Internet connection, unable to provide an OpenStreetMap backdrop. It would be useful to consider how a similar capability could be used to provide coverage in these areas of descending importance:

• NW Scotland
• All UK Waters
• Europe
• Global coverage

I guess some options are:

• GeoTIFFs of UK Admiralty Charts
• UK/Other Coastlines expressed in GeoJSON
• Server-based solution such as OpenStreetMap

Sample analysis task #

• Pick a track (primary) that is present for > 50% of the day
• Randomly apply a dummy sensor activation / deactivation on that track (probably something like a ~1 hour cycle? Can vary to make a more interesting dataset...)
• Assume the sensor has a range of X, select all the other tracks within range of X of primary while the sensor is active (again, can vary X to make the dataset more interesting)
• Produce summary statistics and visualisations of the above. For example, a plot of the # tracks in range as a function of time, a histogram of distances from selected tracks to the primary or a spatial plot that lets you select each interaction. Focus would be on things you wouldn't do in debrief.
• Test the process works by repeating using different days
• An illustration of the concept the above could be used for is in Fisheries Protection. A platform may be suspected of Dark Fishing, and analysts wish to inspect the platforms interactions with others - to see if it may be offloading illegal catch. So, they may wish to view a timeline showing periods when other platforms are in vicinity of the suspect one. Clicking on an item on the timeline would show a map-plot of the interaction - to let them assess if the pattern of behaviour shows that catch was being offloaded. A similarly clickable table of CPAs (Closest Points of Approach - that's not the nearest point between two polylines, but a time-sensitive measurements of the closest point the two moving tracks got to each other) could also be useful. Whereas the above description uses periods of sensor coverage, in this example it could be periods of day/night, or periods with suitable sea-state. Fundamentally, there are temporal windows of interest that relate to the data.

Extended analysis task, considering bulk data #

• Fictional high-level requirement. A government agency is interested in catching smugglers around the UK coastline. They have identified a handful of tracks of known smuggling vessels, but wish to identify occasions when other shipping could have passed within some distance of those vessels. A sample range is 2km.
• The agency wishes to provide a quarterly report on this data. In a quarter it can typically get 20 x 24-hour smuggling vessel tracks, with a minute sample time.
• The report should list the number of "close-encounters" for each smuggling vessels, with extended data on time-duration of close-encounter.
• As sample data for this task, 24-hour datasets can be downloaded from: https://marinecadastre.gov/ais/
• In the UK context, 24 hour of data is a 74MB .csv with ~2000 unique vessels in a 9x700,000 table. So, for a quarter of data there is a very large amount of data present.
• As far as possible, the analysts would like the above process to be automated, so they are free to concentrate on the "close-encounters", and not spend time collating the data
• It is accepted that this bulk-data challenge may introduce new technological challenges/requirements.

Prioritised subsequent tasks #

• Continue with examples of the ‘vessels within X distance of a specific vessel’ analysis, including tidying up and sharing outputs, plus adding other visualisations such as timelines
• Extend ‘vessels within X distance…’ example to deal with ‘periods of interest’ (eg. just during night time etc)
• SQL implementation of Closest Point of Approach analysis
• Think about and demo offline background mapping for use on non-internet-connected systems
• Demonstrate GUI-based dataframe manipulation in a Jupyter notebook using various tools that we’ve discussed before - and showing how those could be used to do interesting quick analyses and could integrate with the rest of Pepys
• Continuing PR to make pandas support SQLAlchemy 2.0 properly (very early PR at ENH: Initial implementation of support for SQLAlchemy 2.0 in read_sql etc (WIP) pandas-dev/pandas#44794, needs quite a bit more work)
• Create a ‘dashboard-style’ demo where you can interactively choose a vessel, choose distance parameters, view plots of timelines and then click to get maps - all built into a nice little demo application (probably using the tool ‘Panel’)