Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Devops/benchmarks #2276

Merged
merged 38 commits into from
Apr 14, 2023
Merged

Devops/benchmarks #2276

merged 38 commits into from
Apr 14, 2023

Conversation

artgish
Copy link
Collaborator

@artgish artgish commented Apr 6, 2023

🚀 🚀 Pull Request

Checklist:

  • My code follows the style guidelines of this project and the Contributing document
  • I have commented my code, particularly in hard-to-understand areas
  • I have kept the coverage-rate up
  • I have performed a self-review of my own code and resolved any problems
  • I have checked to ensure there aren't any other open Pull Requests for the same change
  • I have described and made corresponding changes to the relevant documentation
  • New and existing unit tests pass locally with my changes

Changes

@artgish artgish added the run-benchmarks Run Deep Lake benchmark action label Apr 6, 2023
@github-actions github-actions bot removed the run-benchmarks Run Deep Lake benchmark action label Apr 6, 2023
@github-actions
Copy link

github-actions bot commented Apr 6, 2023 •
edited
Loading

Benchmark Report linux_cpu (Passed)

Benchmark Check Status Performance Diff Current Result Latest Result
image/cifar10
deeplake image/cifar10 local 50000 ingest_serial None None None None Passed 25.05069% 14.376077203000023 19.18107416699968
deeplake image/cifar10 local 50000 ingest_parallel 8 None None None Passed 40.00822% 21.340236330999687 35.571937448001336
deeplake image/cifar10 local 50000 ingest_parallel 16 None None None Passed 40.1029% 21.58566853100092 36.03792113199961
deeplake image/cifar10 local 50000 sequential_read None None None None In Threshold -0.25919% 4.893269641999723 4.880617341999823
deeplake image/cifar10 local 50000 shuffled_read None None None None Passed 0.24331% 5.625268948999292 5.6389908519995515
deeplake_cpp image/cifar10 local 50000 query None None None None Passed 8.93113% 0.41970433999995294 0.4608558889995038
deeplake image/cifar10 local 50000 query None None None None In Threshold -1.98931% 2.5885537060003116 2.5380602610002825
deeplake_python image/cifar10 local 50000 dataloader_iteration 4 4 True None In Threshold -6.42459% 15.745214505999684 14.794713019999108
deeplake_cpp image/cifar10 local 50000 dataloader_iteration 4 4 True None In Threshold -7.83343% 41.27922841800137 38.28054784300002
deeplake_python image/cifar10 local 50000 dataloader_iteration 4 4 False None In Threshold -8.85754% 24.56978533199981 22.570587486001386
deeplake_cpp image/cifar10 local 50000 dataloader_iteration 4 4 False None In Threshold -9.84889% 28.731134568000925 26.155143314999805
deeplake_python image/cifar10 local 50000 dataloader_iteration 8 4 True None Failed -13.72515% 26.08507681500123 22.936946240999532
deeplake_cpp image/cifar10 local 50000 dataloader_iteration 8 4 True None Failed -13.50069% 42.786389216000316 37.697034479999274
deeplake_python image/cifar10 local 50000 dataloader_iteration 8 4 False None Failed -13.11749% 35.08209801399971 31.013864685000954
deeplake_cpp image/cifar10 local 50000 dataloader_iteration 8 4 False None Failed -10.46904% 28.53791102499963 25.833399451999867
deeplake_python image/cifar10 local 50000 dataloader_iteration 16 4 True None Failed -14.27253% 46.44159294800011 40.64107725099893
deeplake_cpp image/cifar10 local 50000 dataloader_iteration 16 4 True None Failed -13.21701% 43.48370596600034 38.407396555998275
deeplake_python image/cifar10 local 50000 dataloader_iteration 16 4 False None Failed -14.00005% 56.073253307000414 49.18703785100115
deeplake_cpp image/cifar10 local 50000 dataloader_iteration 16 4 False None In Threshold -9.988% 29.184702507998736 26.534437161000824
deeplake_python image/cifar10 local 50000 dataloader_iteration 4 8 True None In Threshold -6.62378% 14.638886425000237 13.729483551998783
deeplake_cpp image/cifar10 local 50000 dataloader_iteration 4 8 True None In Threshold -6.10091% 30.292692378001448 28.550825795999117
deeplake_python image/cifar10 local 50000 dataloader_iteration 4 8 False None In Threshold -7.33389% 18.540477327998815 17.27365425300013
deeplake_cpp image/cifar10 local 50000 dataloader_iteration 4 8 False None In Threshold -8.40748% 17.08218771599968 15.757385450999209
deeplake_python image/cifar10 local 50000 dataloader_iteration 8 8 True None Failed -12.0479% 24.842972051999823 22.17174381499899
deeplake_cpp image/cifar10 local 50000 dataloader_iteration 8 8 True None In Threshold -9.2355% 30.603320223000992 28.015913945000648
deeplake_python image/cifar10 local 50000 dataloader_iteration 8 8 False None Failed -12.33979% 28.92598876000011 25.748655232999226
deeplake_cpp image/cifar10 local 50000 dataloader_iteration 8 8 False None Failed -12.19215% 17.349363839000944 15.46396566399926
deeplake_python image/cifar10 local 50000 dataloader_iteration 16 8 True None Failed -14.1495% 45.44866734400057 39.81503772599899
deeplake_cpp image/cifar10 local 50000 dataloader_iteration 16 8 True None Failed -12.29698% 32.23743355499937 28.707295349999185
deeplake_python image/cifar10 local 50000 dataloader_iteration 16 8 False None Failed -14.10447% 49.65966538199973 43.52122874499946
deeplake_cpp image/cifar10 local 50000 dataloader_iteration 16 8 False None Failed -14.7883% 18.352094234000106 15.987769945000764
deeplake image/cifar10 s3 50000 ingest_serial None None None None Passed 17.09577% 19.892960871999094 23.99511197100037
deeplake image/cifar10 s3 50000 ingest_parallel 8 None None None Passed 24.98407% 34.286326763000034 45.70540029199947
deeplake image/cifar10 s3 50000 ingest_parallel 16 None None None Passed 19.11541% 41.384528828999464 51.164910073999636
deeplake image/cifar10 s3 50000 sequential_read None None None None Passed 0.32124% 4.955372758999147 4.971343782000986
deeplake image/cifar10 s3 50000 shuffled_read None None None None In Threshold -0.35757% 5.91920794699945 5.898123158998715
deeplake_cpp image/cifar10 s3 50000 query None None None None Passed 8.07031% 0.7703795270008413 0.8380077599995275
deeplake image/cifar10 s3 50000 query None None None None In Threshold -5.63748% 2.9192622110003867 2.7634685280008853
deeplake_python image/cifar10 s3 50000 dataloader_iteration 4 4 True None In Threshold -6.97309% 21.25165987500077 19.86635564200151
deeplake_cpp image/cifar10 s3 50000 dataloader_iteration 4 4 True None In Threshold -6.90544% 45.267867332000606 42.34384368300016
deeplake_python image/cifar10 s3 50000 dataloader_iteration 4 4 False None In Threshold -9.9451% 29.456486641000083 26.791996346000815
deeplake_cpp image/cifar10 s3 50000 dataloader_iteration 4 4 False None Failed -11.42122% 29.064950527999827 26.085647760999564
deeplake_python image/cifar10 s3 50000 dataloader_iteration 8 4 True None In Threshold -4.74818% 31.56974897500004 30.138707628000702
deeplake_cpp image/cifar10 s3 50000 dataloader_iteration 8 4 True None In Threshold -2.6308% 43.138187431000915 42.03240446199925
deeplake_python image/cifar10 s3 50000 dataloader_iteration 8 4 False None Failed -10.07036% 42.37328893900121 38.4965508589994
deeplake_cpp image/cifar10 s3 50000 dataloader_iteration 8 4 False None Failed -10.4762% 29.11102149999897 26.350488662001226
deeplake_python image/cifar10 s3 50000 dataloader_iteration 16 4 True None Failed -13.37383% 54.96031686800052 48.4770793689986
deeplake_cpp image/cifar10 s3 50000 dataloader_iteration 16 4 True None In Threshold -3.74282% 44.67452010499983 43.06276261499988
deeplake_python image/cifar10 s3 50000 dataloader_iteration 16 4 False None Failed -10.54811% 65.69871529499869 59.42998385200008
deeplake_cpp image/cifar10 s3 50000 dataloader_iteration 16 4 False None Failed -10.54383% 30.06477302300118 27.197151719999965
deeplake_python image/cifar10 s3 50000 dataloader_iteration 4 8 True None Failed -10.67861% 18.466252733998772 16.684572132999165
deeplake_cpp image/cifar10 s3 50000 dataloader_iteration 4 8 True None In Threshold -7.32122% 30.395441085998755 28.321928990999368
deeplake_python image/cifar10 s3 50000 dataloader_iteration 4 8 False None Failed -4511.91271% 24.029911622001237 0.5210368309999467
deeplake_python image/cifar10 s3 50000 dataloader_iteration 8 8 True None In Threshold -9.89479% 29.83315945400136 27.147016205999535
deeplake_cpp image/cifar10 s3 50000 dataloader_iteration 8 8 True None In Threshold -8.64218% 30.937219751998782 28.476252443000703
deeplake_python image/cifar10 s3 50000 dataloader_iteration 8 8 False None Failed -11331.35874% 35.87388629399902 0.3138214990012784
deeplake_python image/cifar10 s3 50000 dataloader_iteration 16 8 True None In Threshold -7.92136% 52.91531111099903 49.03135667099923
deeplake_cpp image/cifar10 s3 50000 dataloader_iteration 16 8 True None In Threshold -9.66567% 32.26954315600051 29.425383113999487
deeplake_python image/cifar10 s3 50000 dataloader_iteration 16 8 False None Failed -11.59534% 59.14736549400004 53.00164906499958
deeplake_cpp image/cifar10 s3 50000 dataloader_iteration 16 8 False None Failed -11.71436% 18.75920922899968 16.792124517000047

Missing checks on benchmarks

Benchmark Check Exist Only Result
deeplake_cpp image/cifar10 s3 50000 dataloader_iteration 4 8 False None Current benchmark 17.40293
deeplake_cpp image/cifar10 s3 50000 dataloader_iteration 8 8 False None Current benchmark 17.92648
deeplake image/mnist local 60000 ingest_serial None None None None Latest benchmark 21.40110922200074
deeplake image/mnist local 60000 ingest_parallel 8 None None None Latest benchmark 38.2804976339994
deeplake image/mnist local 60000 ingest_parallel 16 None None None Latest benchmark 38.713216134001414
deeplake image/mnist local 60000 sequential_read None None None None Latest benchmark 5.446666945999823
deeplake image/mnist local 60000 shuffled_read None None None None Latest benchmark 6.309780621000755
deeplake_cpp image/mnist local 60000 query None None None None Latest benchmark 0.375264956999672
deeplake image/mnist local 60000 query None None None None Latest benchmark 3.0430263100006414
deeplake_python image/mnist local 60000 dataloader_iteration 4 4 True None Latest benchmark 15.107722798000395
deeplake_cpp image/mnist local 60000 dataloader_iteration 4 4 True None Latest benchmark 66.28329441400092
deeplake_python image/mnist local 60000 dataloader_iteration 4 4 False None Latest benchmark 26.64673062300062
deeplake_cpp image/mnist local 60000 dataloader_iteration 4 4 False None Latest benchmark 29.447973604999788
deeplake_python image/mnist local 60000 dataloader_iteration 4 8 True None Latest benchmark 13.544329350001135
deeplake_cpp image/mnist local 60000 dataloader_iteration 4 8 True None Latest benchmark 54.729399492000084
deeplake_python image/mnist local 60000 dataloader_iteration 4 8 False None Latest benchmark 19.801134620000084
deeplake_cpp image/mnist local 60000 dataloader_iteration 4 8 False None Latest benchmark 17.52540468100051
deeplake image/mnist s3 60000 ingest_serial None None None None Latest benchmark 24.325467934999324
deeplake image/mnist s3 60000 ingest_parallel 8 None None None Latest benchmark 48.12197381899932
deeplake image/mnist s3 60000 ingest_parallel 16 None None None Latest benchmark 53.54186603299968
deeplake image/mnist s3 60000 sequential_read None None None None Latest benchmark 5.6137041870006215
deeplake image/mnist s3 60000 shuffled_read None None None None Latest benchmark 6.694982147000701
deeplake_cpp image/mnist s3 60000 query None None None None Latest benchmark 0.715580359001251
deeplake image/mnist s3 60000 query None None None None Latest benchmark 3.143883613000071
deeplake_python image/mnist s3 60000 dataloader_iteration 4 4 True None Latest benchmark 18.932363650999832
deeplake_cpp image/mnist s3 60000 dataloader_iteration 4 4 True None Latest benchmark 70.38965857200128
deeplake_python image/mnist s3 60000 dataloader_iteration 4 4 False None Latest benchmark 29.550384703999953
deeplake_cpp image/mnist s3 60000 dataloader_iteration 4 4 False None Latest benchmark 29.761008830999344
deeplake_python image/mnist s3 60000 dataloader_iteration 4 8 True None Latest benchmark 17.151316482000766
deeplake_cpp image/mnist s3 60000 dataloader_iteration 4 8 True None Latest benchmark 54.65102654399925
deeplake_python image/mnist s3 60000 dataloader_iteration 4 8 False None Latest benchmark 0.8516905540000153
deeplake_python image/mnist local 60000 dataloader_iteration 8 4 True None Latest benchmark 24.047526665999612
deeplake_cpp image/mnist local 60000 dataloader_iteration 8 4 True None Latest benchmark 66.13946447200033
deeplake_python image/mnist local 60000 dataloader_iteration 8 4 False None Latest benchmark 35.66114377899976
deeplake_cpp image/mnist local 60000 dataloader_iteration 8 4 False None Latest benchmark 29.563216243999705
deeplake_python image/mnist local 60000 dataloader_iteration 8 8 True None Latest benchmark 22.498651889998655
deeplake_cpp image/mnist local 60000 dataloader_iteration 8 8 True None Latest benchmark 54.50635330300065
deeplake_python image/mnist local 60000 dataloader_iteration 8 8 False None Latest benchmark 28.571537768999406
deeplake_cpp image/mnist local 60000 dataloader_iteration 8 8 False None Latest benchmark 17.648980459000086
deeplake_python image/mnist s3 60000 dataloader_iteration 8 4 True None Latest benchmark 28.251174833998448
deeplake_cpp image/mnist s3 60000 dataloader_iteration 8 4 True None Latest benchmark 70.37812412200037
deeplake_python image/mnist s3 60000 dataloader_iteration 8 4 False None Latest benchmark 41.4680380789996
deeplake_cpp image/mnist s3 60000 dataloader_iteration 8 4 False None Latest benchmark 30.32402558099966
deeplake_python image/mnist s3 60000 dataloader_iteration 8 8 True None Latest benchmark 27.922217180999723
deeplake_cpp image/mnist s3 60000 dataloader_iteration 8 8 True None Latest benchmark 54.76312492600118
deeplake_python image/mnist s3 60000 dataloader_iteration 8 8 False None Latest benchmark 0.5536464899996645
deeplake_python image/mnist local 60000 dataloader_iteration 16 4 True None Latest benchmark 41.29873505999967
deeplake_cpp image/mnist local 60000 dataloader_iteration 16 4 True None Latest benchmark 66.83211291899897
deeplake_python image/mnist local 60000 dataloader_iteration 16 4 False None Latest benchmark 53.59578287400109
deeplake_cpp image/mnist local 60000 dataloader_iteration 16 4 False None Latest benchmark 30.465686352001285
deeplake_python image/mnist local 60000 dataloader_iteration 16 8 True None Latest benchmark 40.46507379800096
deeplake_cpp image/mnist local 60000 dataloader_iteration 16 8 True None Latest benchmark 55.23979386700012
deeplake_python image/mnist local 60000 dataloader_iteration 16 8 False None Latest benchmark 46.90449547300159
deeplake_cpp image/mnist local 60000 dataloader_iteration 16 8 False None Latest benchmark 18.12031542600016
deeplake_python image/mnist s3 60000 dataloader_iteration 16 4 True None Latest benchmark 49.462569153000004
deeplake_cpp image/mnist s3 60000 dataloader_iteration 16 4 True None Latest benchmark 71.36992229099997
deeplake_python image/mnist s3 60000 dataloader_iteration 16 4 False None Latest benchmark 62.05046746900007
deeplake_cpp image/mnist s3 60000 dataloader_iteration 16 4 False None Latest benchmark 31.084536488000595
deeplake_python image/mnist s3 60000 dataloader_iteration 16 8 True None Latest benchmark 48.39537374500105
deeplake_cpp image/mnist s3 60000 dataloader_iteration 16 8 True None Latest benchmark 56.22819435400015
deeplake_python image/mnist s3 60000 dataloader_iteration 16 8 False None Latest benchmark 0.8386543779997737
deeplake image/coco local 2000 ingest_serial None None None None Latest benchmark 7.805639640000663
deeplake image/coco local 2000 ingest_parallel 8 None None None Latest benchmark 21.761387361000743
deeplake image/coco local 2000 ingest_parallel 16 None None None Latest benchmark 21.835015671998917
deeplake image/coco local 2000 sequential_read None None None None Latest benchmark 7.09931239099933
deeplake image/coco local 2000 shuffled_read None None None None Latest benchmark 8.034958087000632
deeplake_cpp image/coco local 2000 query None None None None Latest benchmark 0.4207058899992262
deeplake image/coco local 2000 query None None None None Latest benchmark 0.47993801300071937
deeplake image/coco s3 2000 ingest_serial None None None None Latest benchmark 13.251455151999835
deeplake image/coco s3 2000 ingest_parallel 8 None None None Latest benchmark 2.0468870740005514
deeplake image/coco s3 2000 ingest_parallel 16 None None None Latest benchmark 43.688114950000454
deeplake image/coco s3 2000 sequential_read None None None None Latest benchmark 15.693202835000193
deeplake image/coco s3 2000 shuffled_read None None None None Latest benchmark 64.309448127
deeplake_cpp image/coco s3 2000 query None None None None Latest benchmark 0.6936434459985321
deeplake image/coco s3 2000 query None None None None Latest benchmark 0.5415183829991292

@artgish artgish added the run-benchmarks Run Deep Lake benchmark action label Apr 6, 2023
@github-actions github-actions bot removed the run-benchmarks Run Deep Lake benchmark action label Apr 6, 2023
@github-actions
Copy link

github-actions bot commented Apr 6, 2023 •
edited
Loading

Benchmark Report linux_gpu (Passed)

Benchmark Check Status Performance Diff Current Result Latest Result
image/cifar10
deeplake image/cifar10 local 50000 ingest_parallel 16 None None None Passed 32.8412586252998% 36.03073318899999 53.65010190999999
deeplake image/cifar10 s3 50000 ingest_parallel 16 None None None Passed 96.78309865014968% 2.0264878150001095 62.99502516900009
image/mnist
deeplake image/mnist local 60000 ingest_parallel 16 None None None Passed 31.520427626344826% 41.32686001700017 60.34917944799997

@artgish artgish added the run-benchmarks Run Deep Lake benchmark action label Apr 10, 2023
@github-actions github-actions bot removed the run-benchmarks Run Deep Lake benchmark action label Apr 10, 2023
@artgish artgish added the run-benchmarks Run Deep Lake benchmark action label Apr 10, 2023
@github-actions github-actions bot removed the run-benchmarks Run Deep Lake benchmark action label Apr 10, 2023
@github-actions
Copy link

github-actions bot commented Apr 10, 2023 •
edited
Loading

Uses cases benchmarks report

DeepLake Status flagship/train_nano flagship/train_t5 mmdet_benchmarks samplers coco_linked_upload
latest 221.92 562.069 ERROR ERROR
sha/088388 Failed 242.618 (-9.33%) 662.437 (-17.86%) ERROR ERROR ERROR

@artgish artgish added the run-benchmarks Run Deep Lake benchmark action label Apr 11, 2023
@github-actions github-actions bot removed the run-benchmarks Run Deep Lake benchmark action label Apr 11, 2023
@artgish artgish added the run-benchmarks Run Deep Lake benchmark action label Apr 12, 2023
@github-actions github-actions bot removed the run-benchmarks Run Deep Lake benchmark action label Apr 12, 2023
@artgish artgish added run-benchmarks Run Deep Lake benchmark action and removed run-benchmarks Run Deep Lake benchmark action labels Apr 14, 2023
@github-actions github-actions bot removed the run-benchmarks Run Deep Lake benchmark action label Apr 14, 2023
@artgish artgish added the run-benchmarks Run Deep Lake benchmark action label Apr 14, 2023
@github-actions github-actions bot removed the run-benchmarks Run Deep Lake benchmark action label Apr 14, 2023
@artgish artgish added the run-benchmarks Run Deep Lake benchmark action label Apr 14, 2023
@artgish artgish added run-benchmarks Run Deep Lake benchmark action and removed run-benchmarks Run Deep Lake benchmark action labels Apr 14, 2023
@github-actions github-actions bot removed the run-benchmarks Run Deep Lake benchmark action label Apr 14, 2023
@artgish artgish added the run-benchmarks Run Deep Lake benchmark action label Apr 14, 2023
@github-actions github-actions bot removed the run-benchmarks Run Deep Lake benchmark action label Apr 14, 2023
@artgish artgish added the run-benchmarks Run Deep Lake benchmark action label Apr 14, 2023
@github-actions github-actions bot removed the run-benchmarks Run Deep Lake benchmark action label Apr 14, 2023
@artgish artgish added the run-benchmarks Run Deep Lake benchmark action label Apr 14, 2023
@github-actions github-actions bot removed the run-benchmarks Run Deep Lake benchmark action label Apr 14, 2023
@artgish artgish marked this pull request as ready for review April 14, 2023 15:54
@tatevikh tatevikh merged commit be746e8 into main Apr 14, 2023
@tatevikh tatevikh deleted the devops/benchmarks branch April 14, 2023 17:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@AbhinavTuli AbhinavTuli Awaiting requested review from AbhinavTuli

@tatevikh tatevikh Awaiting requested review from tatevikh

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@artgish @tatevikh