Skip to content

refactor(models): implement the new data model#322

Merged
frgfm merged 74 commits intomainfrom
datamodel
Jun 4, 2024
Merged

refactor(models): implement the new data model#322
frgfm merged 74 commits intomainfrom
datamodel

Conversation

@frgfm
Copy link
Copy Markdown
Member

@frgfm frgfm commented May 2, 2024

Following up on #304, this PR introduces the following modifications:

  • data model
    UML diagram

  • update all routes

  • update dependencies, the end to end test, the CI

  • simplify docker orchestration by using localstack by default

  • noticeable changes: the camera aren't "users" anymore, but an admin can create an unexpiring token for cameras (that means they can use a created token, but don't have credentials to log in). This reduces API calls & problems with creds

The only things missing are:

  • Updating the tests
  • Updating the client

In the following PRs, this is what will be added:

  • wildfire: aggregation of corresponding detection along time periods & specific geographic areas.
  • region/scope : limit what users can read depending on their spatial scope
  • Webhooks: to enable easily subscribable webhooks for detection creation, and their confirmation.

Any feedback is welcome!

@frgfm frgfm added this to the 0.2.0 milestone May 2, 2024
@frgfm frgfm self-assigned this May 2, 2024
@codecov
Copy link
Copy Markdown

codecov bot commented May 31, 2024

Codecov Report

Attention: Patch coverage is 88.74296% with 60 lines in your changes missing coverage. Please review.

Project coverage is 88.00%. Comparing base (767be30) to head (44506bf).

Files Patch % Lines
src/app/crud/base.py 76.36% 13 Missing ⚠️
src/app/db.py 59.37% 13 Missing ⚠️
src/app/api/api_v1/endpoints/login.py 68.18% 7 Missing ⚠️
src/app/main.py 76.00% 6 Missing ⚠️
src/app/api/api_v1/endpoints/users.py 86.48% 5 Missing ⚠️
src/app/services/storage.py 54.54% 5 Missing ⚠️
src/app/api/api_v1/endpoints/detections.py 94.54% 3 Missing ⚠️
src/app/core/config.py 94.44% 3 Missing ⚠️
src/app/api/dependencies.py 95.45% 2 Missing ⚠️
src/app/services/telemetry.py 60.00% 2 Missing ⚠️
... and 1 more
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #322      +/-   ##
==========================================
- Coverage   94.54%   88.00%   -6.54%     
==========================================
  Files          63       28      -35     
  Lines        1594      642     -952     
==========================================
- Hits         1507      565     -942     
+ Misses         87       77      -10     
Flag Coverage Δ
backend 87.47% <88.56%> (?)
client 95.34% <94.11%> (?)
unittests ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@frgfm
Copy link
Copy Markdown
Member Author

frgfm commented May 31, 2024

Here are the last edits:

  • azimuth field moved from camera table to detection
  • updated the client
  • updated all the tests

The test suite is considerably faster than before 😅

Suggestions for review considering the size of the PR (most of the diff is from the poetry.lock though):

  • check the UML in the PR description to address data model concerns (as mentioned, follow-up PRs will polish this, the heavylifting has been done in this PR)
  • pull the branch,& run the instructions to get the backend up (the docker orchestration has been greatly improved, everything is up and running even offline & fully working), and check the routes in the swagger
  • unfortunately, for the sake of time, you'll have to trust the migration script, the test validity from the CI results
  • make sure to specify if you see major incompatibilities with this datamodel, this is the important part

Things to add to prevent feature reduction compared to the previous version:

  • refine routes for the platform features
  • organization: add a dynamic scope for access management (admin have read/write access everywhere, agent have partial read/write over their organization resources, users have read access over their organization resources)
  • webhooks: implement the notification mechanism when a detection is created
  • wildfire: later on, implement a structured way to access temporal data about wildfires.

@frgfm frgfm requested review from a team, MateoLostanlen, RonanMorgan, blenzi and fe51 May 31, 2024 18:49
RonanMorgan
RonanMorgan previously approved these changes Jun 3, 2024
Copy link
Copy Markdown
Member Author

@frgfm frgfm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @RonanMorgan! Not sure about the sleep in the make though? I thought we implemented healthcheck and add the --wait so that we don't need to wait after that

Or do you still see persisting problems?

Comment thread Makefile Outdated
docker compose -f docker-compose.dev.yml down
-poetry export -f requirements.txt --without-hashes --with test --output requirements.txt
-docker compose -f docker-compose.dev.yml up -d --build --wait
-sleep 30
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't the whole point of the --wait and the healthcheck that we don't need to sleep? 😄

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still have many issues with the database initialization by alembic if I don't add the sleep :/

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's not consistent, it happens one time out of two

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

really? I think we need to investigate that in another PR, sleep is at best a flaky solution :/

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes or we can push a solution and find a better one later :)
since it seems that there is not so much volunteer currently we can revert if you want but it seems to me that it's better to have something which works even if it not super clean

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My debugging mind feels like I'd actually like to have people with problems so we can find the source. But I agree that I didn't expect you to have any, I consistently get everything up & running from the get go 😅

Feel free to open a PR after that to fix that bug (I imagine you have it on the other commands as well then?)

Comment thread src/app/crud/base.py Outdated
async def get_by(self, field_name: str, val: Union[str, int], strict: bool = False) -> Union[ModelType, None]:
statement: Select = select(self.model).where(getattr(self.model, field_name) == val)
results = await self.session.execute(statement=statement)
results = await self.session.exec(statement=statement) # type: ignore
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also let's no ignore all typing issues and be specific if we ignore any
Not sure why it was worth changing this line, since we're now ignore mypy on this line? Perhaps safer to address this in a more specific PR (if the only CI problem is with mypy temporarily, I think it's quite acceptable)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh yes sorry for that I forgot th precise the mypy error to ignore

Comment thread Makefile Outdated
docker compose -f docker-compose.dev.yml up -d --build --wait
docker compose exec -T backend pytest --cov=app
docker compose -f docker-compose.dev.yml down
-poetry export -f requirements.txt --without-hashes --with test --output requirements.txt
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

btw, why are there leading hyphens here?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

leading hyphens are really useful to launch the "docker compose down" even if there are any issue in the commands before

Copy link
Copy Markdown
Member Author

@frgfm frgfm Jun 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh thanks for teaching me, I felt like I didn't know something there!
Mind opening a PR after that one to adjust this on all relevant make commands?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes ok no pb !

@frgfm frgfm merged commit 9a91519 into main Jun 4, 2024
@frgfm frgfm deleted the datamodel branch June 4, 2024 22:27
@frgfm
Copy link
Copy Markdown
Member Author

frgfm commented Aug 24, 2024

cf #304

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants