feat(recorder): profiling and granular recording #24683
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ankush
requested review from
a team and
surajshetty3416
and removed request for
a team
ankush
force-pushed
the
record_and_profile
branch
3 times, most recently
from
f6259ea
to
303b1a9
Compare
ankush
changed the title
ankush
changed the title
ankush
force-pushed
the
record_and_profile
branch
from
303b1a9
to
b4eaf56
Compare
ankush
force-pushed
the
record_and_profile
branch
from
b4eaf56
to
31f010e
Compare
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## develop #24683 +/- ##
===========================================
- Coverage 61.96% 61.35% -0.62%
===========================================
Files 787 788 +1
Lines 75223 75133 -90
Branches 6432 6397 -35
===========================================
- Hits 46613 46097 -516
+ Misses 24891 24759 -132
- Partials 3719 4277 +558
Flags with carried forward coverage won't be shown. Click here to find out more. |
ankush
force-pushed
the
record_and_profile
branch
from
31f010e
to
ccf17da
Compare
|
Also suggest adding sql table filtering, |
|
How so? Roughly:
|
WIP: - [x] Basic working feature - [ ] Make this optional, this has insanely high overhead. - [ ] Specify requests/function filter to profile/record. This will allow better recording in production sites. - [ ] Make SQL profiling optional too
Configurable behaviours: - Record SQL queries - Explain queries - Record stack frames - Run profiler - Filter some requests
use system timezone to avoid showing incorrect timestamp
ankush
force-pushed
the
record_and_profile
branch
from
ccf17da
to
9107627
Compare
ankush
added a commit
that referenced
this pull request
Feb 5, 2024
…24742) * feat: profile requests using recorder WIP: - [x] Basic working feature - [ ] Make this optional, this has insanely high overhead. - [ ] Specify requests/function filter to profile/record. This will allow better recording in production sites. - [ ] Make SQL profiling optional too (cherry picked from commit 3c18334) * feat: configurable recording Configurable behaviours: - Record SQL queries - Explain queries - Record stack frames - Run profiler - Filter some requests (cherry picked from commit e494cb4) * feat: Jobs/Request filtering (cherry picked from commit 009b06d) * fix: recorder timezone bug use system timezone to avoid showing incorrect timestamp (cherry picked from commit 9107627) --------- Co-authored-by: Ankush Menat <ankush@frappe.io>
frappe-pr-bot
pushed a commit
that referenced
this pull request
Feb 6, 2024
# [15.13.0](v15.12.0...v15.13.0) (2024-02-06) ### Bug Fixes * Allow int fields to be unique ([#24750](#24750)) ([#24756](#24756)) ([6c5ea2f](6c5ea2f)) * Avoid enqueueing during install ([#24679](#24679)) ([#24682](#24682)) ([7825a72](7825a72)) * Check if header/footer html exists before trying to extract ([03fc5b1](03fc5b1)) * check is_translatable link ([#24739](#24739)) ([#24747](#24747)) ([e21e2ca](e21e2ca)) * clear sitemap cache periodically ([#24676](#24676)) ([282e0e6](282e0e6)) * console import progress off-by-one ([#24777](#24777)) ([#24787](#24787)) ([e6ad6e8](e6ad6e8)) * Custom Script runs twice ([5d7d7f7](5d7d7f7)) * Dashboard view realtime error ([#24698](#24698)) ([#24699](#24699)) ([a2a766b](a2a766b)) * data import table UI fix ([064aca2](064aca2)) * date format & pre commit test ([9726c46](9726c46)) * depends on ([9777fe3](9777fe3)) * description ([fe7ed4b](fe7ed4b)) * Disappearing letterhead header in pdf ([e932958](e932958)) * do not allow to set if_owner & report perm together ([290b7e8](290b7e8)) * don't notify links if not public ([019c223](019c223)) * enable syntax highlighting for `PythonExpression` code fields ([#24669](#24669)) ([#24671](#24671)) ([4bbfaa7](4bbfaa7)) * Enqueue deletion of dynamic link after comitting ([#24675](#24675)) ([9fbe981](9fbe981)) * Extract header/footer html into `content` to simplify things ([64102a7](64102a7)) * formatting ([05a6ad8](05a6ad8)) * formatting ([5ba9be8](5ba9be8)) * hide number counter for new document view shortcut ([bb46d23](bb46d23)) * **js:** Handle hidden virtual fields ([#24405](#24405)) ([#24692](#24692)) ([ea82c81](ea82c81)) * **login:** Escape translated strings ([#24431](#24431)) ([#24701](#24701)) ([73c66ce](73c66ce)) * lower default retention periods ([#24697](#24697)) ([#24705](#24705)) ([a155f9e](a155f9e)) * make rate_limiter respect multitenancy ([#24634](#24634)) ([d25bfd9](d25bfd9)) * Make sure sitemap respects robot_txt ([10b583b](10b583b)) * never show virtual fields in list view (backport [#24666](#24666)) ([#24668](#24668)) ([45e2683](45e2683)) * no of rows displayed based on report type ([db4d36f](db4d36f)) * outgoing email account handlng (backport [#24656](#24656)) ([#24657](#24657)) ([4a59a01](4a59a01)) * redirect and open new doctype dialog if route to /doctype/new ([fec821a](fec821a)) * Retry contact update if it fails due to conflict ([#24654](#24654)) ([#24655](#24655)) ([22aa5d3](22aa5d3)) * Set default for search result limit if empty ([#24713](#24713)) ([#24715](#24715)) ([474feb8](474feb8)) * short circuit private files perm check ([e6d7ffe](e6d7ffe)) * show folders in Google Drive Picker ([0d99ef8](0d99ef8)), closes [#23096](#23096) * Show proper error message for prepared report failure ([#24733](#24733)) ([#24737](#24737)) ([99676af](99676af)) * Skip fulltext indexes during sync ([#24728](#24728)) ([#24734](#24734)) ([8e078f3](8e078f3)) * suggestion ([63764a9](63764a9)) * title link check ([#24731](#24731)) ([#24735](#24735)) ([5c006cd](5c006cd)) * Translate Letterhead information string ([8dfa972](8dfa972)) * typeerror in onboarding_tours.js ([5ae79be](5ae79be)) * Use `TEXT` type for Webhook URLs to support large URLs ([#24763](#24763)) ([f31a3ce](f31a3ce)) * **UX:** improve server script message ([#24770](#24770)) ([5d88275](5d88275)) * virtual fields are never writable ([#24693](#24693)) ([#24696](#24696)) ([d020771](d020771)) * **website:** Handle virtual DocTypes webview routing ([#24445](#24445)) ([#24743](#24743)) ([bed66ca](bed66ca)) ### Features * allow extending site config with a hook ([ffed7bf](ffed7bf)) * Letterhead scripts ([9b296cd](9b296cd)) * **recorder:** profiling and granular recording (backport [#24683](#24683)) ([#24742](#24742)) ([fd04617](fd04617)) * show doctype description on list title hover (backport [#24612](#24612)) ([#24769](#24769)) ([5a0e20b](5a0e20b)) * show doctype description on workspace link hover ([#24598](#24598)) ([#24789](#24789)) ([6107338](6107338)) * use_first_day_of_period ([1fcb105](1fcb105)) ### Performance Improvements * optional faster perm check for files ([3e4a344](3e4a344))
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
New recorder configuration to configure what you want to record and process
You can now use cProfile (similar to
%prun -s cumulativefrom ipython)closes #23981
docs: https://frappeframework.com/docs/user/en/profiling