Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Datastore to Firestore Migration #142951

Open
24 of 44 tasks
keyonghan opened this issue Feb 6, 2024 · 0 comments
Open
24 of 44 tasks

Datastore to Firestore Migration #142951

keyonghan opened this issue Feb 6, 2024 · 0 comments
Assignees
@stuartmorgan
Labels
P2 Important issues not at the top of the work list team-infra Owned by Infrastructure team

Comments

@keyonghan
Copy link
Contributor

keyonghan commented Feb 6, 2024 •
edited by stuartmorgan
Loading

This is a tracking issue to migrat Datastore to Firestore.
@keyonghan keyonghan self-assigned this Feb 6, 2024
@keyonghan keyonghan added team-infra Owned by Infrastructure team P2 Important issues not at the top of the work list labels Feb 6, 2024
@keyonghan keyonghan mentioned this issue Feb 6, 2024
Merged
auto-submit bot pushed a commit to flutter/cocoon that referenced this issue Feb 6, 2024
@keyonghan
Initialize Firestore client and service (#3471)
80cbb4d 
The very first step to support Firestore. Part of flutter/flutter#142951

This PR:
1) Creates a client support with customized header support
2) Adds a basic FirestoreService based on the client
@keyonghan keyonghan mentioned this issue Feb 7, 2024
Merged
auto-submit bot pushed a commit to flutter/cocoon that referenced this issue Feb 8, 2024
@keyonghan
Start writing new commits/tasks to Firestore (#3472)
62d724e 
This PR
1) starts writing new commit/tasks to the new Firestore database
2) is a no-op to existing workflow. Will catch any exception without breaking existing logic.
3) puts generic logics (like documents to writes, comits/tasks to documents, etc) in the firestore service
4) adds extensive tests for each logic added. 

Part of flutter/flutter#142951.
This was referenced Feb 8, 2024
Closed
Merged
Merged
auto-submit bot pushed a commit to flutter/cocoon that referenced this issue Feb 8, 2024
@keyonghan
Remove builderNumber when adding new tasks (#3474)
6aa1ca2 
Follow up of #3473.

It turns out that Firestore doesn't accept null/empty values for `integerValue`.
This PR removes the field when adding new tasks. The `builderNumber` field will be added when updating task statuses, where the real build number will be available until then.

Fixes: flutter/flutter#143187
More context: flutter/flutter#142951
This was referenced Feb 16, 2024
Merged
Merged
Merged
auto-submit bot pushed a commit to flutter/cocoon that referenced this issue Feb 20, 2024
@keyonghan
Supports Firestore task update for /api/update-task-status (#3495)
a2dfa9e 
Part of flutter/flutter#142951
auto-submit bot pushed a commit to flutter/cocoon that referenced this issue Feb 20, 2024
@keyonghan
Support multiple records for retries on a same task (#3494)
83a44f0 
This PR adds support to save multiple entries for a task retry. Old logic overrides task statuses on a single task, which fails to track statuses.

This PR:
1) makes it customizable to collect `attempts` info
2) adds `current_attempt` tag to make it easy to track `attempts` info.
3) handles `attempt` increase when retry is triggered.

Part of flutter/flutter#142951.
This was referenced Feb 20, 2024
Merged
Merged
auto-submit bot pushed a commit to flutter/cocoon that referenced this issue Feb 21, 2024
@keyonghan
Update Firestore task when mulitple reruns exist (#3506)
0e74e8c 
This is a follow up of #3495.

This PR:
1) updates the latest task status correctly when multiple records exist
2) uses const variables to avoid potential typos

Part of flutter/flutter#142951
@keyonghan keyonghan mentioned this issue Feb 21, 2024
Merged
auto-submit bot pushed a commit to flutter/cocoon that referenced this issue Feb 21, 2024
@keyonghan
Update Firestore task documents when back filling (#3510)
0c814a7 
Part of flutter/flutter#142951

This PR starts updating task documents in Firestore when batch back filling tasks.
auto-submit bot pushed a commit to flutter/cocoon that referenced this issue Feb 21, 2024
@keyonghan
Update Firestore when reseting a prod task (#3505)
e85150d 
Part of flutter/flutter#142951

This PR:
1) inserts new task document to Firestore when retried manually
2) fixes `trigger_type` tag: instead of overriding manual case as `trigger_type: auto`, we now distinguish them with
2.1) `trigger_type: manual_retry` for manual retry
2.2) `trigger_type: auto_retry` for auto retry
@keyonghan keyonghan mentioned this issue Feb 22, 2024
Merged
auto-submit bot pushed a commit to flutter/cocoon that referenced this issue Feb 23, 2024
@keyonghan
Update task documents in Firestore when vacuuming stale tasks (#3511)
e75d187 
Part of flutter/flutter#142951
@keyonghan keyonghan mentioned this issue Feb 23, 2024
Merged
auto-submit bot pushed a commit to flutter/cocoon that referenced this issue Feb 23, 2024
@keyonghan
Create firestore commit model and reference it everywhere (#3512)
16b1937 
Help with flutter/flutter#142951
@keyonghan keyonghan mentioned this issue Feb 23, 2024
Merged
auto-submit bot pushed a commit to flutter/cocoon that referenced this issue Feb 23, 2024
@keyonghan
Updates github gold status to Firestore (#3513)
857e09a 
Part of flutter/flutter#142951

This PR:
1) starts updating Github Gold Status to Firestore
2) updates status in a try block to avoid breaking prod workflow
3) creates a corresponding Firestore model
4) updates `documentsToWrites` to have default `exists=null` to support generic document writes.
@keyonghan keyonghan mentioned this issue Feb 27, 2024
Merged
auto-submit bot pushed a commit to flutter/cocoon that referenced this issue Feb 28, 2024
@keyonghan
Write dart-internal tasks to Firestore (#3526)
886463b 
Part of flutter/flutter#142951 for releases branch support.
sealesj pushed a commit to sealesj/cocoon that referenced this issue Mar 7, 2024
@keyonghan @sealesj
Write github build status to Firestore (flutter#3540)
4ed004c 
Part of flutter/flutter#142951

This PR:

- starts updating Github Build Status to Firestore
- updates status in a try block to avoid breaking prod workflow
- creates a corresponding Firestore model
sealesj pushed a commit to sealesj/cocoon that referenced this issue Mar 7, 2024
@keyonghan @sealesj
Add a wrapper for Firestore query to simplify logics (flutter#3553)
670d68f 
Part of flutter/flutter#142951.

This PR:
1) adds a query wrapper to simplify logics
2) switch existing commitTask query to use the new wrapper
@keyonghan keyonghan mentioned this issue Mar 8, 2024
Merged
auto-submit bot pushed a commit to flutter/cocoon that referenced this issue Mar 8, 2024
@keyonghan
Switch github gold status API toFirestore (#3555)
4db41a4 
Part of flutter/flutter#142951.

The github gold status API has been querying and writing updates to Datastore.
#3513 started writing updates to Firestore.

This PR:
1) starts querying from Firestore
2) supports both datastore/Firestore queries and writes
3) updates all unit tests to cover Firestore logics
@keyonghan keyonghan mentioned this issue Mar 13, 2024
Merged
auto-submit bot pushed a commit to flutter/cocoon that referenced this issue Mar 13, 2024
@keyonghan
Query recent commits from firestore (#3572)
84be1e8 
Part of flutter/flutter#142951.

This is to prepare switching get-build-status queries to firestore.

This PR:
1) add function queryRecentCommits
2) add query option: `orders` and `limit`.
@keyonghan keyonghan mentioned this issue Mar 14, 2024
Merged
auto-submit bot pushed a commit to flutter/cocoon that referenced this issue Mar 18, 2024
@keyonghan
Calculate tree build status based on Firestore task statuses (#3575)
02192ce 
Part of flutter/flutter#142951

This PR:
1) keeps existing function `retrieveCommitStatus` and creates a new `retrieveCommitStatusFirestore` for `get_build_status` API. This is to avoid affecting other APIs.
2) keeps existing class `CommitStatus` and creates a new `CommitTasksStatus` to support firestore case, preparing to deprecate the `stage` field from the old Task status.
3) maps to the firestore commit/task fields from old Datastore models and updates all related tests.
@keyonghan keyonghan mentioned this issue Mar 18, 2024
Merged
auto-submit bot pushed a commit to flutter/cocoon that referenced this issue Mar 19, 2024
@keyonghan
Push build status based on Firestore (#3582)
95d6986 
Part of flutter/flutter#142951.

The github build status API has been querying and writing updates to Datastore.

This PR:

- starts querying from Firestore
- supports both datastore/Firestore queries and writes
- updates all unit tests to cover Firestore logics
@keyonghan keyonghan mentioned this issue Mar 19, 2024
Merged
auto-submit bot pushed a commit to flutter/cocoon that referenced this issue Mar 19, 2024
@keyonghan
Support Firestore user data for postsubmit build (#3583)
3db8d54 
Part of flutter/flutter#142951.

To migrate postsubmit build API fully to firestore, we need to support user data servicing firestore data models. This PR adds corresponding document names.
@keyonghan keyonghan mentioned this issue Mar 19, 2024
Merged
auto-submit bot pushed a commit to flutter/cocoon that referenced this issue Mar 19, 2024
@keyonghan
Switch to use firestore task for postsubmit_luci_subscription API (#3584
f817add 
)

Part of flutter/flutter#142951

Following #3583, this PR:
1) switches logics to use firestore task
2) keeps querying/writing task status for both Datastore and Firestore, in case fallback happens
3) updates all unit tests
4) removes the condition when `taskKey` doesn't exist (seems all builds are enforced with the `userData`, and confirmed no call for the past week)
@keyonghan keyonghan mentioned this issue Mar 22, 2024
Merged
auto-submit bot pushed a commit to flutter/cocoon that referenced this issue Mar 25, 2024
@keyonghan
Use Firestore data when reseting prod task and rerunning a build (#3589)
1314886 
Part of flutter/flutter#142951

This PR:
1) supports `taskDocumentName` arg for reset-prod-task API. This is to prepare frontend logic switched to Firestore
2) uses Firestore data when checking rerun, instead of Datastore data
3) moves `taskFailStatusSet` to Task models
4) moves Firestore logics out of try/catch block to break API if failing
This was referenced Apr 2, 2024
Closed
Merged
auto-submit bot pushed a commit to flutter/cocoon that referenced this issue Apr 10, 2024
@keyonghan
Add firestore data model protos (#3632)
9ccd1e7 
This is to prepare frontend firestore migration for Cocoon dashboard.

Part of flutter/flutter#142951.
One can view #3619 for a quick reference how we are gonna to use these protos.

This PR adds:

- commit_firestore.proto, a replacement of commit.proto for Datastore case
- task_firestore.proto, a replacement of task.proto for Datastore case
- commit_tasks_status.proto, a replacement of commit_status.proto for Datastore case

Note: *.pb.dart, *.pbenum.dart, *.pbjson.dart, *.pbserver.dart are auto generated based on the proto.
@keyonghan keyonghan mentioned this issue Apr 10, 2024
Merged
auto-submit bot pushed a commit to flutter/cocoon that referenced this issue Apr 11, 2024
@keyonghan
Add logics to fetch commit status from Firestore (#3634)
bbfd8e8 
This is part of flutter/flutter#142951

This PR adds logics to fetch commit tasks status from Firestore based on newly added Firestore data model protos. After this lands, next step is to switch the build logic to reference this PR's changes from Datastore.

This PR is a no-op change by this point.
@keyonghan keyonghan mentioned this issue Apr 11, 2024
Merged
auto-submit bot pushed a commit to flutter/cocoon that referenced this issue Apr 15, 2024
@keyonghan
Add API api/public/get-status-firestore (#3635)
ab40d73 
Part of flutter/flutter#142951.

This PR adds api entry point for `api/public/get-status-firestore`. This is prepare frontend migration to Firestore.

This change is a no-op, as the frontend is still calling `api/public/get-status`.
@keyonghan keyonghan mentioned this issue Apr 17, 2024
Merged
auto-submit bot pushed a commit to flutter/cocoon that referenced this issue Apr 17, 2024
@keyonghan
Add BuildList field to ease frontend logics (#3669)
f1f6621 
Frontend needs `BuildList` to handle case of multiple reruns. This PR updates the returned results from.

```
statuses:
  commit,
  tasks
```

to 

```
statuses:
  Commit,
  Tasks
    Task
    BuildList
```

Part of flutter/flutter#142951.
@keyonghan keyonghan mentioned this issue Apr 23, 2024
Closed
@christopherfujino christopherfujino mentioned this issue Apr 29, 2024
Open
@christopherfujino christopherfujino mentioned this issue Jun 12, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@stuartmorgan stuartmorgan

Labels
P2 Important issues not at the top of the work list team-infra Owned by Infrastructure team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@stuartmorgan @keyonghan