Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: proof-of-concept elastic-backed PVGroup + archiver status retrieval #1

Merged
merged 15 commits into from
Dec 1, 2020

Conversation

klauer
Copy link
Contributor

@klauer klauer commented Nov 18, 2020

  • Proof-of-concept for elastic database-backed PVGroups.
  • Queries the actual archiver (or the included sim archiver management interface)
  • Maps archiver status keys onto PV names and PVGroup attribute names (in a not ideal, but not terrible fashion)

@klauer
Copy link
Contributor Author

klauer commented Nov 18, 2020

Showing that the values are updating at regular intervals (30s, way faster than Murali requested for now):

Output of camonitor (big list of PVs):
ARCH:pscaa02:AvgTimeSpentByFreeSpaceChecksInEtl1To2SPerRun 2020-11-17 16:10:30.251348 2.14
ARCH:pscaa02:AvgTimeSpentByAppendtoetlappenddataInEtl1To2SPerRun 2020-11-17 16:10:30.251487 55.92
ARCH:pscaa02:AvgTimeSpentByMarkfordeletionInEtl1To2SPerRun 2020-11-17 16:10:30.251626 1.44
ARCH:pscaa01:Eventrate         2020-11-17 16:11:01.259787 1875.03
ARCH:pscaa01:Secondsconsumedbywritter 2020-11-17 16:11:01.260123 0.27619
ARCH:pscaa01:Datarate          2020-11-17 16:11:01.260288 105857
ARCH:pscaa02:Datarategbperyear 2020-11-17 16:11:01.260501 3284.17
ARCH:pscaa02:Secondsconsumedbywritter 2020-11-17 16:11:01.260649 0.280012
ARCH:pscaa02:Datarate          2020-11-17 16:11:01.260789 111820
ARCH:pscaa01:EventRateInEventsPerSec 2020-11-17 16:11:01.826228 1875.03
ARCH:pscaa01:DataRateInBytesPerSec 2020-11-17 16:11:01.826487 105857
ARCH:pscaa01:BenchmarkWritingAtEventsPerSec 2020-11-17 16:11:01.826654 67889
ARCH:pscaa01:TotalNumberOfEtl1To2RunsSoFar 2020-11-17 16:11:01.826863 82
ARCH:pscaa01:AverageTimeSpentInEtl1To2SPerRun 2020-11-17 16:11:01.827015 53.77
ARCH:pscaa01:AvgTimeSpentByGetetlstreamsInEtl1To2SPerRun 2020-11-17 16:11:01.827237 5.3
ARCH:pscaa01:AvgTimeSpentByFreeSpaceChecksInEtl1To2SPerRun 2020-11-17 16:11:01.827409 1.58
ARCH:pscaa01:AvgTimeSpentByAppendtoetlappenddataInEtl1To2SPerRun 2020-11-17 16:11:01.827553 45.68
ARCH:pscaa01:AvgTimeSpentByMarkfordeletionInEtl1To2SPerRun 2020-11-17 16:11:01.827693 1.17
ARCH:pscaa02:DataRateInBytesPerSec 2020-11-17 16:11:02.548561 111820
ARCH:pscaa02:AverageTimeSpentInEtl1To2SPerRun 2020-11-17 16:11:02.548926 72.89
ARCH:pscaa01:Eventrate         2020-11-17 16:11:33.462620 1875.06
ARCH:pscaa01:Datarategbperyear 2020-11-17 16:11:33.462917 3109.06
ARCH:pscaa01:Secondsconsumedbywritter 2020-11-17 16:11:33.463123 0.276189
ARCH:pscaa01:Datarate          2020-11-17 16:11:33.463281 105858
ARCH:pscaa01:Totaletlruns1     2020-11-17 16:11:33.463453 82
ARCH:pscaa02:Eventrate         2020-11-17 16:11:33.463615 1897.7
ARCH:pscaa02:Secondsconsumedbywritter 2020-11-17 16:11:33.463767 0.280011
ARCH:pscaa02:Datarate          2020-11-17 16:11:33.463945 111820
ARCH:pscaa01:EventRateInEventsPerSec 2020-11-17 16:11:34.030638 1875.05
ARCH:pscaa01:DataRateInBytesPerSec 2020-11-17 16:11:34.030923 105857
ARCH:pscaa01:DataRateInGbPerYear 2020-11-17 16:11:34.031111 3109.05
ARCH:pscaa01:BenchmarkWritingAtEventsPerSec 2020-11-17 16:11:34.031271 67890
ARCH:pscaa01:AverageTimeSpentInEtl1To2SPerRun 2020-11-17 16:11:34.031474 53.78
ARCH:pscaa02:EventRateInEventsPerSec 2020-11-17 16:11:34.664153 1897.7
ARCH:pscaa02:DataRateInBytesPerSec 2020-11-17 16:11:34.664437 111820
ARCH:pscaa02:DataRateInGbPerYear 2020-11-17 16:11:34.664622 3284.17
ARCH:pscaa02:BenchmarkWritingAtEventsPerSec 2020-11-17 16:11:34.664813 67772.4
ARCH:pscaa01:Datarategbperyear 2020-11-17 16:12:05.498953 3109.05
ARCH:pscaa01:Secondsconsumedbywritter 2020-11-17 16:12:05.499738 0.276189
ARCH:pscaa01:Datarate          2020-11-17 16:12:05.500270 105857
ARCH:pscaa02:Secondsconsumedbywritter 2020-11-17 16:12:05.500985 0.280011
ARCH:pscaa02:Datarate          2020-11-17 16:12:05.501601 111820
ARCH:pscaa02:Timeforoveralletlinseconds1 2020-11-17 16:12:05.501819 5977
ARCH:pscaa01:DataRateInBytesPerSec 2020-11-17 16:12:06.084376 105857
ARCH:pscaa01:DataRateInGbPerYear 2020-11-17 16:12:06.084687 3109.04
ARCH:pscaa01:BenchmarkWritingAtEventsPerSec 2020-11-17 16:12:06.084859 67890
ARCH:pscaa02:DataRateInBytesPerSec 2020-11-17 16:12:06.704431 111820
ARCH:pscaa02:BenchmarkWritingAtEventsPerSec 2020-11-17 16:12:06.704715 67772.2
ARCH:pscaa02:ApproximateTimeTakenByLastJobInEtl1To2S 2020-11-17 16:12:06.704974 1
ARCH:pscaa02:AvgTimeSpentByGetetlstreamsInEtl1To2SPerRun 2020-11-17 16:12:06.705170 13.34
ARCH:pscaa01:Eventrate         2020-11-17 16:12:37.565924 1875.04
ARCH:pscaa01:Datarategbperyear 2020-11-17 16:12:37.566247 3109.03
ARCH:pscaa01:Secondsconsumedbywritter 2020-11-17 16:12:37.566432 0.276189
ARCH:pscaa01:Datarate          2020-11-17 16:12:37.566602 105857
ARCH:pscaa02:Eventrate         2020-11-17 16:12:37.566801 1897.69
ARCH:pscaa02:Secondsconsumedbywritter 2020-11-17 16:12:37.566970 0.280011
ARCH:pscaa02:Datarate          2020-11-17 16:12:37.567143 111820
ARCH:pscaa01:DataRateInBytesPerSec 2020-11-17 16:12:38.117118 105857
ARCH:pscaa01:DataRateInGbPerYear 2020-11-17 16:12:38.117414 3109.05
ARCH:pscaa01:BenchmarkWritingAtEventsPerSec 2020-11-17 16:12:38.117581 67890.1
ARCH:pscaa02:EventRateInEventsPerSec 2020-11-17 16:12:38.980971 1897.69
ARCH:pscaa02:DataRateInBytesPerSec 2020-11-17 16:12:38.981737 111820
ARCH:pscaa02:BenchmarkWritingAtEventsPerSec 2020-11-17 16:12:38.981945 67771.9
ARCH:pscaa02:AverageTimeSpentInEtl1To2SPerRun 2020-11-17 16:12:38.982189 72.9
ARCH:pscaa01:Eventrate         2020-11-17 16:13:09.830016 1875.02
ARCH:pscaa01:Datarategbperyear 2020-11-17 16:13:09.830424 3109
ARCH:pscaa01:Secondsconsumedbywritter 2020-11-17 16:13:09.830668 0.276189
ARCH:pscaa01:Datarate          2020-11-17 16:13:09.830908 105856
ARCH:pscaa01:Timeforoveralletlinseconds1 2020-11-17 16:13:09.831240 4410
ARCH:pscaa02:Datarategbperyear 2020-11-17 16:13:09.831515 3284.16
ARCH:pscaa02:Secondsconsumedbywritter 2020-11-17 16:13:09.831746 0.280012
ARCH:pscaa02:Datarate          2020-11-17 16:13:09.832012 111820
ARCH:pscaa01:EventRateInEventsPerSec 2020-11-17 16:13:10.358163 1875.01
ARCH:pscaa01:DataRateInBytesPerSec 2020-11-17 16:13:10.358432 105855
ARCH:pscaa01:DataRateInGbPerYear 2020-11-17 16:13:10.358601 3108.99
ARCH:pscaa01:BenchmarkWritingAtEventsPerSec 2020-11-17 16:13:10.358784 67888.6
ARCH:pscaa01:ApproximateTimeTakenByLastJobInEtl1To2S 2020-11-17 16:13:10.359015 1
ARCH:pscaa02:EventRateInEventsPerSec 2020-11-17 16:13:10.975870 1897.7
ARCH:pscaa02:BenchmarkWritingAtEventsPerSec 2020-11-17 16:13:10.976177 67772.1
ARCH:pscaa01:Eventrate         2020-11-17 16:13:41.795153 1875.01
ARCH:pscaa01:Secondsconsumedbywritter 2020-11-17 16:13:41.795423 0.276189
ARCH:pscaa01:Datarate          2020-11-17 16:13:41.795583 105855
ARCH:pscaa02:Secondsconsumedbywritter 2020-11-17 16:13:41.795786 0.280011
ARCH:pscaa02:Datarate          2020-11-17 16:13:41.795932 111819
ARCH:pscaa01:EventRateInEventsPerSec 2020-11-17 16:13:42.368216 1875
ARCH:pscaa01:DataRateInBytesPerSec 2020-11-17 16:13:42.368493 105855
ARCH:pscaa01:DataRateInGbPerYear 2020-11-17 16:13:42.368655 3108.98
ARCH:pscaa01:BenchmarkWritingAtEventsPerSec 2020-11-17 16:13:42.368808 67888.3
ARCH:pscaa01:AvgTimeSpentByGetetlstreamsInEtl1To2SPerRun 2020-11-17 16:13:42.369020 5.31
ARCH:pscaa02:EventRateInEventsPerSec 2020-11-17 16:13:43.000558 1897.69
ARCH:pscaa02:DataRateInBytesPerSec 2020-11-17 16:13:43.000834 111820
ARCH:pscaa02:DataRateInGbPerYear 2020-11-17 16:13:43.001088 3284.16
ARCH:pscaa02:BenchmarkWritingAtEventsPerSec 2020-11-17 16:13:43.001281 67771.8
ARCH:pscaa01:Secondsconsumedbywritter 2020-11-17 16:14:13.838767 0.276188
ARCH:pscaa01:Datarate          2020-11-17 16:14:13.839071 105856
ARCH:pscaa02:Eventrate         2020-11-17 16:14:13.839274 1897.68
ARCH:pscaa02:Datarategbperyear 2020-11-17 16:14:13.839427 3284.15
ARCH:pscaa02:Secondsconsumedbywritter 2020-11-17 16:14:13.839572 0.280011
ARCH:pscaa02:Datarate          2020-11-17 16:14:13.839713 111819
ARCH:pscaa02:Timeforoveralletlinseconds1 2020-11-17 16:14:13.839888 5978
ARCH:pscaa01:DataRateInBytesPerSec 2020-11-17 16:14:14.393138 105855
ARCH:pscaa01:BenchmarkWritingAtEventsPerSec 2020-11-17 16:14:14.393438 67888.4
ARCH:pscaa02:EventRateInEventsPerSec 2020-11-17 16:14:15.011949 1897.68
ARCH:pscaa02:DataRateInBytesPerSec 2020-11-17 16:14:15.012273 111819
ARCH:pscaa02:DataRateInGbPerYear 2020-11-17 16:14:15.012486 3284.15
ARCH:pscaa02:BenchmarkWritingAtEventsPerSec 2020-11-17 16:14:15.012709 67771.6
ARCH:pscaa02:AvgTimeSpentByGetetlstreamsInEtl1To2SPerRun 2020-11-17 16:14:15.012964 13.35
ARCH:pscaa01:Eventrate         2020-11-17 16:14:45.855926 1875
ARCH:pscaa01:Datarategbperyear 2020-11-17 16:14:45.856211 3108.99
ARCH:pscaa01:Secondsconsumedbywritter 2020-11-17 16:14:45.856389 0.276188
ARCH:pscaa01:Datarate          2020-11-17 16:14:45.856541 105855
ARCH:pscaa02:Eventrate         2020-11-17 16:14:45.856721 1897.67
ARCH:pscaa02:Secondsconsumedbywritter 2020-11-17 16:14:45.856873 0.280011
ARCH:pscaa02:Datarate          2020-11-17 16:14:45.857013 111819
ARCH:pscaa01:EventRateInEventsPerSec 2020-11-17 16:14:46.400720 1874.99
ARCH:pscaa01:DataRateInBytesPerSec 2020-11-17 16:14:46.401431 105855
ARCH:pscaa01:BenchmarkWritingAtEventsPerSec 2020-11-17 16:14:46.401921 67888.1
ARCH:pscaa01:AverageTimeSpentInEtl1To2SPerRun 2020-11-17 16:14:46.402429 53.79
ARCH:pscaa02:EventRateInEventsPerSec 2020-11-17 16:14:47.087347 1897.67
ARCH:pscaa02:DataRateInBytesPerSec 2020-11-17 16:14:47.088031 111819
ARCH:pscaa02:BenchmarkWritingAtEventsPerSec 2020-11-17 16:14:47.088624 67771.3
ARCH:pscaa02:AverageTimeSpentInEtl1To2SPerRun 2020-11-17 16:14:47.089525 72.91
ARCH:pscaa02:ApproximateTimeTakenByLastJobInEtl1To2S 2020-11-17 16:14:47.089992 2

@klauer
Copy link
Contributor Author

klauer commented Dec 1, 2020

caproto updates PVs + writes to elasticsearch, grafana reads from elasticsearch:

image

Separate indices for each archiver instance to make the fields the same. Indices now date-based as well:

image

@klauer klauer merged commit 4b6bfc3 into pcdshub:master Dec 1, 2020
@klauer klauer deleted the db_backed branch December 1, 2020 18:46
@klauer klauer mentioned this pull request Dec 1, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant