Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Encode timestamp fraction according to the Proto3 JSON Specification. #8035

Closed
wants to merge 2 commits into from

Conversation

MarkDuckworth
Copy link
Contributor

The SDK encodes a string representation of a timestamp for use as the key value in client side indexes. For this, and other reason, the SDK and Firestore backend must stringify timestamps identically.

Fixes: #8031

Copy link

changeset-bot bot commented Feb 22, 2024

🦋 Changeset detected

Latest commit: 370a5e0

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages
Name Type
@firebase/firestore Patch
firebase Patch
@firebase/firestore-compat Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Copy link
Contributor

github-actions bot commented Feb 22, 2024

Changeset File Check ✅

  • No modified packages are missing from the changeset file.
  • No changeset formatting errors detected.

@MarkDuckworth MarkDuckworth requested a review from a team as a code owner February 22, 2024 01:14
@google-oss-bot
Copy link
Contributor

google-oss-bot commented Feb 22, 2024

Size Report 1

Affected Products

  • @firebase/firestore

    TypeBase (9fa0e9f)Merge (373c2b5)Diff
    browser375 kB375 kB+101 B (+0.0%)
    esm5360 kB360 kB+116 B (+0.0%)
    main577 kB577 kB+135 B (+0.0%)
    module375 kB375 kB+101 B (+0.0%)
    react-native375 kB375 kB+101 B (+0.0%)
  • @firebase/firestore-lite

    TypeBase (9fa0e9f)Merge (373c2b5)Diff
    browser109 kB109 kB+101 B (+0.1%)
    esm5106 kB106 kB+116 B (+0.1%)
    main150 kB150 kB+135 B (+0.1%)
    module109 kB109 kB+101 B (+0.1%)
    react-native109 kB110 kB+101 B (+0.1%)
  • bundle

    14 size changes

    TypeBase (9fa0e9f)Merge (373c2b5)Diff
    firestore (CSI Auto Indexing Disable and Delete)268 kB268 kB+101 B (+0.0%)
    firestore (CSI Auto Indexing Enable)268 kB268 kB+101 B (+0.0%)
    firestore (Persistence)303 kB303 kB+101 B (+0.0%)
    firestore (Query Cursors)246 kB246 kB+101 B (+0.0%)
    firestore (Query)243 kB243 kB+101 B (+0.0%)
    firestore (Read data once)231 kB232 kB+101 B (+0.0%)
    firestore (Read Write w Persistence)321 kB321 kB+101 B (+0.0%)
    firestore (Realtime updates)234 kB234 kB+101 B (+0.0%)
    firestore (Transaction)211 kB211 kB+101 B (+0.0%)
    firestore (Write data)211 kB211 kB+101 B (+0.0%)
    firestore-lite (Query Cursors)89.4 kB89.5 kB+101 B (+0.1%)
    firestore-lite (Query)85.5 kB85.6 kB+101 B (+0.1%)
    firestore-lite (Transaction)86.6 kB86.7 kB+101 B (+0.1%)
    firestore-lite (Write data)71.3 kB71.4 kB+101 B (+0.1%)

  • firebase

    TypeBase (9fa0e9f)Merge (373c2b5)Diff
    firebase-compat.js780 kB780 kB+110 B (+0.0%)
    firebase-firestore-compat.js341 kB341 kB+110 B (+0.0%)
    firebase-firestore-lite.js117 kB117 kB+110 B (+0.1%)
    firebase-firestore.js434 kB435 kB+110 B (+0.0%)

Test Logs

  1. https://storage.googleapis.com/firebase-sdk-metric-reports/QVWIvCcieh.html

@google-oss-bot
Copy link
Contributor

google-oss-bot commented Feb 22, 2024

Size Analysis Report 1

Affected Products

  • @firebase/firestore

    • QueryEndAtConstraint

      Size

      TypeBase (9fa0e9f)Merge (373c2b5)Diff
      size43.5 kB43.6 kB+101 B (+0.2%)
      size-with-ext-deps114 kB114 kB+101 B (+0.1%)
    • QueryFieldFilterConstraint

      Size

      TypeBase (9fa0e9f)Merge (373c2b5)Diff
      size41.4 kB41.5 kB+101 B (+0.2%)
      size-with-ext-deps111 kB112 kB+101 B (+0.1%)
    • QueryStartAtConstraint

      Size

      TypeBase (9fa0e9f)Merge (373c2b5)Diff
      size43.5 kB43.6 kB+101 B (+0.2%)
      size-with-ext-deps114 kB114 kB+101 B (+0.1%)
    • Transaction

      Size

      TypeBase (9fa0e9f)Merge (373c2b5)Diff
      size51.7 kB51.8 kB+101 B (+0.2%)
      size-with-ext-deps122 kB122 kB+101 B (+0.1%)
    • WriteBatch

      Size

      TypeBase (9fa0e9f)Merge (373c2b5)Diff
      size48.4 kB48.5 kB+101 B (+0.2%)
      size-with-ext-deps118 kB119 kB+101 B (+0.1%)
    • addDoc

      Size

      TypeBase (9fa0e9f)Merge (373c2b5)Diff
      size130 kB130 kB+101 B (+0.1%)
      size-with-ext-deps202 kB202 kB+101 B (+0.0%)
    • and

      Size

      TypeBase (9fa0e9f)Merge (373c2b5)Diff
      size42.7 kB42.8 kB+101 B (+0.2%)
      size-with-ext-deps113 kB113 kB+101 B (+0.1%)
    • arrayRemove

      Size

      TypeBase (9fa0e9f)Merge (373c2b5)Diff
      size30.7 kB30.8 kB+101 B (+0.3%)
      size-with-ext-deps101 kB101 kB+101 B (+0.1%)
    • arrayUnion

      Size

      TypeBase (9fa0e9f)Merge (373c2b5)Diff
      size30.7 kB30.8 kB+101 B (+0.3%)
      size-with-ext-deps101 kB101 kB+101 B (+0.1%)
    • deleteDoc

      Size

      TypeBase (9fa0e9f)Merge (373c2b5)Diff
      size121 kB121 kB+101 B (+0.1%)
      size-with-ext-deps193 kB193 kB+101 B (+0.1%)
    • enableIndexedDbPersistence

      Size

      TypeBase (9fa0e9f)Merge (373c2b5)Diff
      size186 kB186 kB+101 B (+0.1%)
      size-with-ext-deps258 kB258 kB+101 B (+0.0%)
    • enableMultiTabIndexedDbPersistence

      Size

      TypeBase (9fa0e9f)Merge (373c2b5)Diff
      size222 kB222 kB+101 B (+0.0%)
      size-with-ext-deps294 kB294 kB+101 B (+0.0%)
    • endAt

      Size

      TypeBase (9fa0e9f)Merge (373c2b5)Diff
      size43.5 kB43.6 kB+101 B (+0.2%)
      size-with-ext-deps114 kB114 kB+101 B (+0.1%)
    • endBefore

      Size

      TypeBase (9fa0e9f)Merge (373c2b5)Diff
      size43.5 kB43.6 kB+101 B (+0.2%)
      size-with-ext-deps114 kB114 kB+101 B (+0.1%)
    • executeWrite

      Size

      TypeBase (9fa0e9f)Merge (373c2b5)Diff
      size120 kB120 kB+101 B (+0.1%)
      size-with-ext-deps192 kB192 kB+101 B (+0.1%)
    • getDoc

      Size

      TypeBase (9fa0e9f)Merge (373c2b5)Diff
      size150 kB150 kB+101 B (+0.1%)
      size-with-ext-deps222 kB222 kB+101 B (+0.0%)
    • getDocFromServer

      Size

      TypeBase (9fa0e9f)Merge (373c2b5)Diff
      size150 kB150 kB+101 B (+0.1%)
      size-with-ext-deps222 kB222 kB+101 B (+0.0%)
    • getDocs

      Size

      TypeBase (9fa0e9f)Merge (373c2b5)Diff
      size152 kB152 kB+101 B (+0.1%)
      size-with-ext-deps224 kB224 kB+101 B (+0.0%)
    • getDocsFromServer

      Size

      TypeBase (9fa0e9f)Merge (373c2b5)Diff
      size151 kB152 kB+101 B (+0.1%)
      size-with-ext-deps223 kB224 kB+101 B (+0.0%)
    • onSnapshot

      Size

      TypeBase (9fa0e9f)Merge (373c2b5)Diff
      size152 kB152 kB+101 B (+0.1%)
      size-with-ext-deps224 kB224 kB+101 B (+0.0%)
    • onSnapshotsInSync

      Size

      TypeBase (9fa0e9f)Merge (373c2b5)Diff
      size142 kB142 kB+101 B (+0.1%)
      size-with-ext-deps214 kB214 kB+101 B (+0.0%)
    • or

      Size

      TypeBase (9fa0e9f)Merge (373c2b5)Diff
      size42.7 kB42.8 kB+101 B (+0.2%)
      size-with-ext-deps113 kB113 kB+101 B (+0.1%)
    • persistentLocalCache

      Size

      TypeBase (9fa0e9f)Merge (373c2b5)Diff
      size183 kB183 kB+101 B (+0.1%)
      size-with-ext-deps255 kB255 kB+101 B (+0.0%)
    • persistentMultipleTabManager

      Size

      TypeBase (9fa0e9f)Merge (373c2b5)Diff
      size219 kB219 kB+101 B (+0.0%)
      size-with-ext-deps291 kB291 kB+101 B (+0.0%)
    • persistentSingleTabManager

      Size

      TypeBase (9fa0e9f)Merge (373c2b5)Diff
      size183 kB183 kB+101 B (+0.1%)
      size-with-ext-deps255 kB255 kB+101 B (+0.0%)
    • query

      Size

      TypeBase (9fa0e9f)Merge (373c2b5)Diff
      size43.0 kB43.1 kB+101 B (+0.2%)
      size-with-ext-deps113 kB113 kB+101 B (+0.1%)
    • runTransaction

      Size

      TypeBase (9fa0e9f)Merge (373c2b5)Diff
      size130 kB130 kB+101 B (+0.1%)
      size-with-ext-deps202 kB202 kB+101 B (+0.1%)
    • setDoc

      Size

      TypeBase (9fa0e9f)Merge (373c2b5)Diff
      size129 kB129 kB+101 B (+0.1%)
      size-with-ext-deps201 kB201 kB+101 B (+0.1%)
    • startAfter

      Size

      TypeBase (9fa0e9f)Merge (373c2b5)Diff
      size43.5 kB43.6 kB+101 B (+0.2%)
      size-with-ext-deps114 kB114 kB+101 B (+0.1%)
    • startAt

      Size

      TypeBase (9fa0e9f)Merge (373c2b5)Diff
      size43.5 kB43.6 kB+101 B (+0.2%)
      size-with-ext-deps114 kB114 kB+101 B (+0.1%)
    • updateDoc

      Size

      TypeBase (9fa0e9f)Merge (373c2b5)Diff
      size130 kB130 kB+101 B (+0.1%)
      size-with-ext-deps202 kB202 kB+101 B (+0.1%)
    • where

      Size

      TypeBase (9fa0e9f)Merge (373c2b5)Diff
      size42.2 kB42.3 kB+101 B (+0.2%)
      size-with-ext-deps112 kB112 kB+101 B (+0.1%)
    • writeBatch

      Size

      TypeBase (9fa0e9f)Merge (373c2b5)Diff
      size132 kB132 kB+101 B (+0.1%)
      size-with-ext-deps204 kB204 kB+101 B (+0.0%)

Test Logs

  1. https://storage.googleapis.com/firebase-sdk-metric-reports/AbqvXMsFx7.html


// Pad the fraction out to 3, 6, or 9 digits
let microStr = ('000000000' + timestamp.nanoseconds).slice(-9);
while (microStr.endsWith('000')) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't endsWith() be startsWith()? By using endsWith(), isn't that loop effectively dividing the fraction by 1000 on each iteration?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The loop trims "000" off the end. So 1.123000 becomes 1.123. This has the same numeric value but allows comparison of the date strings returned by Firestore backend, which also perform this truncation.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about backwards compatibility? Do we need to write a data migration to convert timestamps from the old format to the new format?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are persisting date strings from the backend, which already have the truncated format, so at least for this we don't need a migration. However, I'm running into some issues with edge cases and I think using the Proto3 JSON date format for comparison will not work.

@MarkDuckworth
Copy link
Contributor Author

After testing some edge cases, there are other issues with comparing these Proto3 JSON timestamp values. This approach will not work. Closing this PR for now.

@firebase firebase locked and limited conversation to collaborators Mar 24, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Equality check for timestamp fields fails when caching is active in a web worker.
5 participants