Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Observability reporter - move JSON serialization step #14992

Merged
merged 8 commits into from
Dec 13, 2023
Merged

Observability reporter - move JSON serialization step #14992

merged 8 commits into from
Dec 13, 2023

Conversation

przlada
Copy link
Member

@przlada przlada commented Dec 12, 2023 •
edited

I want to merge this change because it adds a JSON serialization step to observability events creation functions. It aims to achieve the following benefits:

  • Eliminate an extra serialization/deserialization step when storing events in the buffer.
  • Drop the usage of the cPickle library.

While Pickle offers faster serialization than JSON, the simple data structure and small size (max limit of 25 kB) of observability events result in negligible performance impact. Local tests indicate a maximal increase of approximately 0.14 milliseconds per event, with the potential for further reduction through the integration of faster JSON serialization libraries.

Impact

  • New migrations
  • New/Updated API fields or mutations
  • Deprecated API fields or mutations
  • Removed API types, fields, or mutations

Docs

  • Link to documentation:

Pull Request Checklist

  • Privileged queries and mutations are either absent or guarded by proper permission checks
  • Database queries are optimized and the number of queries is constant
  • Database migrations are either absent or optimized for zero downtime
  • The changes are covered by test cases
  • All new fields/inputs/mutations have proper labels added (ADDED_IN_X, PREVIEW_FEATURE, etc.)
  • All migrations have proper dependencies
  • All indexes are added concurrently in migrations
  • All RunSql and RunPython migrations have revert option defined

@db-queries
Copy link

db-queries bot commented Dec 12, 2023 •
edited

Here is the report for cfff945 (saleor:move-observability-events-serialization)
Base comparison is 6816d63.

No differences were found. 

# saleor.graphql.accountbenchmark account
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  addresses for federation query count                                    	          9	          9	              2
  customers query                                                         	         51	         51	              0
  delete staff members                                                    	         41	         41	              0
  query staff user                                                        	         22	         22	              3
  staff create                                                            	         23	         23	              3
  staff update groups and permissions                                     	         36	         36	              5
  users for federation query count                                        	          8	          8	              3

# saleor.graphql.accountbenchmark permission group
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  groups for federation query count                                       	          5	          5	              0
  permission group create                                                 	         25	         25	              0
  permission group delete                                                 	         30	         30	              3
  permission group query                                                  	         14	         14	              0
  permission group update                                                 	         44	         44	              2
  permission group update remove users with manage staff                  	         36	         36	              3

# saleor.graphql.appbenchmarks app extensions
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  app extensions                                                          	         15	         15	              0
  app extensions with filter[filter0]                                     	         14	         14	              0
  app extensions with filter[filter1]                                     	         10	         10	              0
  app extensions with filter[filter2]                                     	         14	         14	              0
  app extensions with filter[filter3]                                     	         10	         10	              0

# saleor.graphql.appbenchmarks apps
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  apps for federation query count                                         	          9	          9	              3
  apps with tokens and webhooks                                           	          9	          9	              0

# saleor.graphql.attributebenchmark attribute
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  attribute translation                                                   	          6	          6	              0
  attribute value translation                                             	         20	         20	              0
  query attribute                                                         	          6	          6	              0
  query attributes                                                        	          8	          8	              0

# saleor.graphql.channelbenchmark channel
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  channels query                                                          	          5	          5	              0

# saleor.graphql.channelmutations channel update
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  channel update mutation remove warehouse                                	         38	         38	              6

# saleor.graphql.checkoutbenchmark checkout mutations
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  add billing address to checkout                                         	         71	         71	             10
  add checkout lines                                                      	         74	         74	             11
  add checkout lines with external shipping                               	        178	        178	             74
  add checkout lines with reservations                                    	        157	        157	             56
  add delivery to checkout                                                	         72	         72	             12
  add shipping to checkout                                                	         82	         82	             15
  checkout email update                                                   	         37	         37	              1
  checkout payment charge                                                 	         51	         51	             14
  checkout shipping address update                                        	         76	         76	              9
  checkout voucher code                                                   	         94	         94	             16
  complete checkout                                                       	        234	        234	             79
  complete checkout preorder                                              	        232	        232	             80
  complete checkout with digital line                                     	        275	        275	             93
  complete checkout with out of stock webhook                             	        236	        236	             79
  complete checkout with single line                                      	        236	        236	             79
  create checkout                                                         	         77	         77	              8
  create checkout with reservations                                       	        136	        136	             23
  customer complete checkout                                              	        247	        247	             85
  customer complete checkout for cc                                       	        222	        222	             74
  update checkout lines                                                   	         67	         67	             11
  update checkout lines with reservations                                 	        163	        163	             67

# saleor.graphql.checkoutbenchmark checkouts
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  staff multiple checkouts                                                	          6	          6	              0

# saleor.graphql.checkoutbenchmark homepage
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  user checkout details                                                   	         57	         57	              5
  user checkout details with tax app                                      	         51	         51	              2

# saleor.graphql.discountbenchmark promotion create
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  promotion create                                                        	         49	         49	              3

# saleor.graphql.discountbenchmark promotion delete
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  promotion delete                                                        	         29	         29	              1

# saleor.graphql.discountbenchmark promotion rule create
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  promotion rule create                                                   	         38	         38	              0

# saleor.graphql.discountbenchmark promotion rule delete
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  promotion rule delete                                                   	         19	         19	              0

# saleor.graphql.discountbenchmark promotion rule update
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  promotion rule update                                                   	         36	         36	              2

# saleor.graphql.discountbenchmark promotion update
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  promotion update                                                        	         33	         33	              1

# saleor.graphql.discountbenchmark promotions
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  promotions querytest promotions query                                   	          9	          9	              0

# saleor.graphql.discountbenchmark sales
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  sales query with channel slug                                           	          9	          9	              0
  sales query without channel slug                                        	          7	          7	              0

# saleor.graphql.discountbenchmark voucher code bulk delete
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  voucher code bulk delete queries                                        	         22	         22	              2

# saleor.graphql.discountbenchmark vouchers
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  vouchers query with channel slug                                        	         20	         20	              0
  vouchers query withot channel slug                                      	         19	         19	              0

# saleor.graphql.giftcardbenchmark gift card mutations
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  create never expiry gift card                                           	         26	         26	              4
  gift card bulk activate by staff                                        	         16	         16	              3
  update gift card                                                        	         36	         36	              4

# saleor.graphql.giftcardbenchmark gift card queries
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  filter gift cards by products                                           	         14	         14	              3
  filter gift cards by tags                                               	         12	         12	              3
  filter gift cards by used by user                                       	         13	         13	              3
  query gift card details                                                 	         14	         14	              3
  query gift cards                                                        	         12	         12	              3

# saleor.graphql.orderbenchmark fulfillment
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  fulfillment query                                                       	          6	          6	              0

# saleor.graphql.orderbenchmark fulfillment refund and return products
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  fulfillment refund products order lines                                 	         61	         61	              1
  fulfillment return products order lines                                 	        105	        105	              8

# saleor.graphql.orderbenchmark order
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  staff multiple draft orders                                             	          7	          7	              0
  staff multiple orders                                                   	          7	          7	              0
  staff order details                                                     	         57	         57	              8
  user order details                                                      	         55	         55	              9

# saleor.graphql.orderbenchmark order bulk create
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  order bulk create                                                       	         21	         21	              2

# saleor.graphql.orderbenchmark order fulfill
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  order fulfill                                                           	         19	         19	              0
  order fulfill with gift cards                                           	         19	         19	              0

# saleor.graphql.orderbenchmark order lines create
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  order lines create                                                      	         94	         94	             17
  order lines create variants on promotion                                	        148	        148	             22

# saleor.graphql.pagebenchmark page type
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  page types for federation query count                                   	          3	          3	              0
  query page type                                                         	         18	         18	              6
  query page types                                                        	         25	         25	             12

# saleor.graphql.paymentbenchmark payment transactions
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  payment transactions                                                    	         54	         54	             20

# saleor.graphql.productbenchmark category
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  categories children                                                     	          3	          3	              0
  categories for federation query count                                   	          3	          3	              0
  category delete                                                         	         63	         63	              9
  category view                                                           	         28	         28	              1

# saleor.graphql.productbenchmark collection
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  collection add products                                                 	         44	         44	              8
  collection bulk delete                                                  	         52	         52	              6
  collection view                                                         	          6	          6	              0
  collections for federation query count                                  	          7	          7	              0
  create collection                                                       	         36	         36	              5
  delete collection                                                       	         48	         48	              9
  remove products from collection                                         	         41	         41	              8
  retrieve collection channel listings                                    	          5	          5	              0

# saleor.graphql.productbenchmark homepage
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  report product sales                                                    	         10	         10	              3
  retrieve product list                                                   	          4	          4	              0

# saleor.graphql.productbenchmark product
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  filter products by attributes                                           	          7	          7	              0
  filter products by boolean attributes                                   	         14	         14	              0
  filter products by gift card                                            	         11	         11	              1
  filter products by numeric attributes                                   	         13	         13	              0
  product create                                                          	         67	         67	              3
  product details                                                         	         42	         42	              1
  product translations                                                    	          5	          5	              0
  products for federation query count                                     	          6	          6	              0
  products media for federation query count                               	          5	          5	              0
  products types for federation query count                               	          2	          2	              0
  retrieve channel listings                                               	         19	         19	              0
  retrieve product attributes                                             	          8	          8	              0
  retrieve product images                                                 	          4	          4	              0
  retrieve product media                                                  	          4	          4	              0
  retrive products with product types and attributes                      	          7	          7	              0
  update product                                                          	         60	         60	              5

# saleor.graphql.productbenchmark product variant channel listing update
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  variant channel listing update                                          	         46	         46	              3

# saleor.graphql.productbenchmark variant
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  product variant bulk create                                             	         65	         65	              1
  product variant create                                                  	         69	         69	              3
  products variants for federation query count                            	          6	          6	              0
  retrieve variant list                                                   	         35	         35	              1
  update product variant                                                  	         67	         67	              5

# saleor.graphql.productbenchmark variant stocks
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  product variants stocks create                                          	         29	         29	              4
  product variants stocks create with single webhook called               	         28	         28	              4
  product variants stocks delete by id                                    	         30	         30	              4
  product variants stocks delete by sku                                   	         30	         30	              4
  product variants stocks delete with out of stock webhook many calls     	         26	         26	              3
  product variants stocks update by sku                                   	         28	         28	              4
  product variants stocks update byid                                     	         28	         28	              4
  query product variants stocks                                           	         10	         10	              0

# saleor.graphql.producttest product sorting attributes
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  sort product not having attribute data                                  	         20	         20	              0

# saleor.graphql.shippingbenchmark shipping methods
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  vouchers query with channel slug                                        	          7	          7	              0
  vouchers query without channel slug                                     	          7	          7	              0

# saleor.graphql.shippingbenchmark shipping zones
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  shipping zones query                                                    	          6	          6	              0

# saleor.graphql.shopbenchmark homepage
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  retrieve shop                                                           	          2	          2	              0

# saleor.graphql.warehousebenchmark stock bulk update
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  stocks bulk update queries count                                        	         28	         28	              4

# saleor.graphql.warehousebenchmark stocks
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  stocks query                                                            	          5	          5	              0

# saleor.graphql.webhookbenchmark webhook events
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  webhooks                                                                	          6	          6	              0

@codecov Codecov
Copy link

codecov bot commented Dec 12, 2023 •
edited

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (d81bc98) 93.81% compared to head (cfff945) 93.88%.
Report is 3 commits behind head on main.

Additional details and impacted files 
@@            Coverage Diff             @@
##             main   #14992      +/-   ##
==========================================
+ Coverage   93.81%   93.88%   +0.06%     
==========================================
  Files         927      928       +1     
  Lines       69532    69655     +123     
  Branches     7992     8015      +23     
==========================================
+ Hits        65232    65394     +162     
+ Misses       2779     2748      -31     
+ Partials     1521     1513       -8

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@przlada przlada requested a review from a team December 12, 2023 11:16
@korycins korycins requested a review from a team December 12, 2023 12:54
@przlada przlada self-assigned this Dec 12, 2023
@przlada przlada merged commit 0df62b3 into main Dec 13, 2023
8 checks passed
@przlada przlada deleted the move-observability-events-serialization branch December 13, 2023 13:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@michal-macioszczyk michal-macioszczyk michal-macioszczyk approved these changes

@korycins korycins korycins approved these changes

@IKarbowiak IKarbowiak IKarbowiak approved these changes

Assignees

@przlada przlada

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@przlada @michal-macioszczyk @korycins @IKarbowiak