Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(github): Don't store updatedAt field in GraphQL cache record #21024

Conversation

zharinov
Copy link
Collaborator

@zharinov zharinov commented Mar 18, 2023

Changes

  • Separate updatedAt and cache data in order to optimize cache I/O
  • Remove updatedAt field
  • Rewrite cache only if any of items has changed

Context

Documentation (please check one with an [x])

  • I have updated the documentation, or
  • No documentation update is required

How I've tested my work (please select one)

I have verified these changes via:

  • Code inspection only, or
  • Newly added/modified unit tests, or
  • No unit tests but ran on a real repository, or
  • Both unit tests + ran on a real repository

@rarkins
Copy link
Collaborator

rarkins commented Mar 18, 2023

What is the expected outcome?

When I do a dry run on this repo two times in a row I still see a lot of package cache sets:

DEBUG: Repository timing splits (milliseconds) (repository=renovatebot/renovate)
       "splits": {"init": 15133, "extract": 3496, "lookup": 14482},
       "total": 33262
DEBUG: Package cache statistics (repository=renovatebot/renovate)
       "get": {"count": 246, "avgMs": 269, "medianMs": 205, "maxMs": 1643},
       "set": {"count": 24, "avgMs": 431, "medianMs": 354, "maxMs": 1561}
DEBUG: http statistics (repository=renovatebot/renovate)
       "urls": {
         "https://api.github.com/graphql (POST,200)": 26,
         "https://api.github.com/repos/renovatebot/.github/contents/default.json (GET,200)": 1
       },
       "hostStats": {"api.github.com": {"requestCount": 27, "requestAvgMs": 1171, "queueAvgMs": 0}},
       "totalRequests": 27

I assume those 24 are for github tags/releases and that they're unnecessary.

@rarkins
Copy link
Collaborator

rarkins commented Mar 18, 2023

My expectation is that when I run on a repo two times in a row that there are no package cache sets on the second run.

@zharinov
Copy link
Collaborator Author

That's because we're still saving new updatedAt value to the package cache each run (it costs about 5ms).

@rarkins
Copy link
Collaborator

rarkins commented Mar 18, 2023

Why do we need to set the updatedAt?

Let's say I asked you not to set it every run. What would we lose?

@zharinov
Copy link
Collaborator Author

Hah, turns out it's the legacy field from the time when we tried to do zero GraphQL queries per run. So the answer is we only lose the potential possibility of doing zero queries, but seems like we don't want it anyway.

@zharinov zharinov changed the title feat(github): Store GraphQL cache data and timestamp at separate keys feat(github): Don't store updatedAt field in GraphQL cache record Mar 18, 2023
viceice
viceice previously approved these changes Mar 18, 2023
Copy link
Member

@viceice viceice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤷‍♂️ don't we have a equal comparison lib already?

@zharinov
Copy link
Collaborator Author

🤦 We do

Copy link
Member

@viceice viceice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, so please use the existing 😁

@zharinov zharinov requested a review from viceice March 18, 2023 14:27
Copy link
Collaborator

@rarkins rarkins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DEBUG: Package cache statistics (repository=renovatebot/renovate)
       "get": {"count": 222, "avgMs": 275, "medianMs": 186, "maxMs": 1669},
       "set": {"count": 0}

@rarkins rarkins enabled auto-merge March 18, 2023 14:29
@rarkins rarkins added this pull request to the merge queue Mar 18, 2023
@rarkins rarkins merged commit 42b0e10 into renovatebot:main Mar 18, 2023
8 checks passed
@rarkins rarkins deleted the feat/graphql-cache-separate-data-and-timestamp branch March 18, 2023 15:20
@renovate-release
Copy link
Collaborator

🎉 This PR is included in version 35.11.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Apr 18, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Avoid package cache set every run for GitHub tags/releases
4 participants