Use GCP code coverage in backend #2108
Conversation
La0
force-pushed
the
ccov-path-gcp
branch
3 times, most recently
from
c740ef6 to
63db25e
Compare
La0
force-pushed
the
ccov-path-gcp
branch
2 times, most recently
from
1400710 to
4808081
Compare
| raise Exception('Path {} not found in report'.format(object_path)) | ||
| report = report['children'][part] | ||
|
|
||
| def _clean_object(obj, base_path, depth=0): |
There was a problem hiding this comment.
I wonder if some of the changes here couldn't be avoided, leaving the onus on the frontend.
E.g. adding the type could be unnecessary, as the frontend can check if there are 'children' itself.
Similarly for path.
There was a problem hiding this comment.
I did that to ease the work on the frontend side, but i'm totally ok with leaving the covdir format as it is.
There was a problem hiding this comment.
Yeah let's do that. At least removing type and path. This way we keep the backend as lean as possible, almost just serving the report file.
| def coverage_for_path(path='', changeset=None): | ||
| def coverage_latest(repository=DEFAULT_REPOSITORY): | ||
| ''' | ||
| List the last 10 reports available on the server |
There was a problem hiding this comment.
The frontend will likely only need the latest
There was a problem hiding this comment.
For now... I did that because it was easy to do now, and could provide a quick <select> on the frontend to display recent variations of a file's coverage
| logger.info('Reports will be stored in {}'.format(self.reports_dir)) | ||
|
|
||
| # Load most recent reports in cache | ||
| for repo in ('mozilla-central', ): |
There was a problem hiding this comment.
Nit: define repositories in a constant at the beginning of the file
| if not chunk: | ||
| break | ||
| output.write(chunk) | ||
|
|
There was a problem hiding this comment.
Should we remove the zstd file now?
There was a problem hiding this comment.
As long as we are on heroku yes.
When we switch to GCP, we will have some FS persistence, and could retain archives as long term storage (and only keep the warm json extracted)
| raise Exception('No report found') | ||
| return results[0] | ||
|
|
||
| def find_closest_report(self, repository, revision): |
There was a problem hiding this comment.
Nit: we might want to keep a consistent naming and either always use changeset or revision
marco-c
left a comment
marco-c
left a comment
There was a problem hiding this comment.
LGTM! Some modifications will be required for getting filters (#1227) to work, but they should be reasonable.
2 participants
This PR replace the
/v2/pathendpoint to use GCP covdir data + a redis cache.Organisational data is stored in redis:
changeset:deadbeef➡️12345)reports➡️{12345: "deadbeef", ...}The coverage data is downloaded from GCP, extracted and stored in a local file path.
⚠️ No coverage data is available in Redis