Fetching past fires via the API

[pechouc]

Hello everyone,

Here is a non-urgent issue related to this one in the Pyro-Platform repository!

In order to display past fires on the map, we are currently reading a local file stored within the repository. The goal would be to remove it and instead fetch this information via the Pyro-API. If I understood well, for now, there is no table dedicated to past fires (which could be useful not only for the platform but also for the data science team typically) and this issue is designed to discuss its addition to the database.

To be more precise about the historic_fires.csv file, it was built out of a raw dataset of satellite fire detections which we preprocessed, notably in this Colab. We added several fields like the department and the "commune" where the fire occurred (as well as the population, area and density of the locality).

We wanted to use the latter to compute the density of population in the area, so as to approximately filter urban fires out of the dataset. However, we have not yet implemented this filtering on the platform and for now, department is the only added field which is really required. I mention this in case we are limited for the number of fields.

Last remark: at some point, I guess we might want to add fires detected by our own devices and acknowledged by firemen to the past fires table in the database.

All that being said, here are my question(s):

• Should we add the historic_fires.csv file as is to the database?
• Or do you prefer that we script the preprocessing steps and use the raw dataset?

Thanks a lot for your help, I hope that everything is clear!

[frgfm]

Hey there 👋

I agree that this should be added in the database, but then there are a few things to decide:

1. If we are to add data in the DB, for clarity, I would suggest adding actual wildfire history. By any chance, do you have any source that could be of help @pechouc? Alternatively, we could populate it using the client manually 🤷‍♂️
2. Now let's say it's available in the DB, what is the scope of your request? all past wildfire events? the ones in a specific region? This is tightly linked with the discussion [API] Fetch alerts metadata from API pyro-platform#38 about what we put in cache, and what each individual user will actually request from the API

Tagging a few extra people @jeanpasquier75 @florianriche @martin1tab @chloeskt, any thoughts?

[pechouc]

1. If we are to add data in the DB, for clarity, I would suggest adding actual wildfire history. By any chance, do you have any source that could be of help @pechouc? Alternatively, we could populate it using the client manually 🤷‍♂️

I am not sure I have fully understood your question but we have indeed built the csv file (that can be found here) from actual wildfires detected by a MODIS satellite. We have added to the original, raw dataset several fields about the localisation of the fire, the type of area in which it took place, etc.

So, to add it in the database, I guess that there are two options: either directly adding the data that is encapsulated in the csv or scripting the whole preprocessing sequence so as to populate the database from the raw data file, similar to what can be downloaded from the MODIS website for instance.

2. Now let's say it's available in the DB, what is the scope of your request? all past wildfire events? the ones in a specific region? This is tightly linked with the discussion pyronear/pyro-platform#38 about what we put in cache, and what each individual user will actually request from the API

For sure, it will ultimately be related to this discussion, especially if alerts raised by our devices and acknowledged by firemen are supposed to enter the database and are displayed as "past fires" on the map after a given time period. But leaving aside this mechanism, the csv file we are currently using is based exclusively on publicly accessible information, so it makes sense - at least before we can refine the process - to give all users of the platform a full access to our database of past fires. If we go for this option, we could have users request and load the whole file without any confidentiality issues.

[frgfm]

Oh alright, I thought it was a mock fire history.

Regarding the database, we have to think about the long-term production case. So we need an updated (or several) source of wildfire with accurate location that we can fetch. If there is an API to request that would be good news!

For the scope, let's restrict this for now to admin, which will be the scope granted to the backend of the platform. Otherwise we won't get things moving forward soon 😅

[jean-pasquier]

From my understanding, we want to store the fire history within our database. This implies to create new routes to the API querying this table, at least:

• GET the history to display on the platform map
• POST new fire
  • in a first milestone, "manually" by an admin
  • afterward, as an enhancement, through the platform (eg once an alert is acknowledged, etc. - to refine)

Am I correct ? 😄

The csv would be loaded to the db using the API client, is that what you meant @frgfm ?

The question is who is allowed to GET the data and is there an access by department/area ? Since the data is open as @pechouc mentioned, I would say no

By "the scope of your request" @frgfm did you refer to any filter on GET request (time range, area, department, etc.) ?

[frgfm]

@jeanpasquier75 yes about the methods

Regarding how we fill the DB, the API client could be a good way indeed

Exactly about the scope of the request

[frgfm]

Alright, time to close this issue! I'll open a PR by the end of the weekend 👌

[frgfm]

@pechouc the feature was just deployed and made available through the client 👌
The entry in the documentation is over here: https://pyronear.org/pyro-api/client.html#pyroclient.client.Client.get_past_events