Repository navigation
Monitor a Collection
You can monitor a collection using the Datahub API to get summaries of recently collected data for the collection or to query specific data. You can troubleshoot a collection by looking at the log files on the collector server.
The DataHub API is documented at http://18.135.72.18/api/v1/.
You must connect to it from an IP address that has been whitelisted for access (e.g. the Oxford VPN). You will also need an API token for access. Please discuss this with your server administrator.
The API documentation describes how to query the list_collections endpoint to get a list of collection that you have access to. You can then get a quick summary of recent data collected using the monitor_collections endpoint. You can use the query endpoint to access the raw data. See the API Documentation for details.
If a collection is not working as expected, then you may need to check the log files on the collector server to troubleshoot the issue. To do this, first ssh into the server.
Then, navigate to the data directory for your collection:
cd ~/mySocialWatcher/data/[collector_name]/[collection_name]
ls -lh
You can look at the simple launch.log file to see a record of when collections were launched and completed.
cat launch.log
To see a detailed log, look into the "logs" directory:
cd logs
ls -lh
Inspect recent log files to look for error messages:
head 20230802_000001.log
tail 20230802_000001.log
cat 20230802_000001.log
Even if the log files are not showing up above, it's worth manually checking the docker logs, as it's possible there's an issue with the software writing out the log files.
docker logs dgg_national