Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failed removing completed task from database #458

Closed
sandreae opened this issue Jul 24, 2023 · 10 comments · Fixed by #461
Closed

Failed removing completed task from database #458

sandreae opened this issue Jul 24, 2023 · 10 comments · Fixed by #461
Labels
bug Something isn't working

Comments

@sandreae
Copy link
Member 

[2023-07-24T13:08:49Z INFO  aquadoggo::replication::manager] Accept inbound replication session with peer 12D3KooWMHjFHCHCQiZDHG5VmTcD4U5SCAyeR9BM82rNBAo7rxWF (4)
[2023-07-24T13:08:49Z INFO  aquadoggo::replication::manager] Accept inbound replication session with peer 12D3KooWAkfqtGjR23jgLg9YmuDqruMoGJqkaH2bxiRbCnjAPjuQ (1)
[2023-07-24T13:08:49Z INFO  aquadoggo::replication::service] Finished replication with peer 12D3KooWKkbEq3owBUBYRCQzruzMskAWt8w2Lqaq7VNGNeSXBXF5 (2)
[2023-07-24T13:08:49Z INFO  aquadoggo::replication::service] Finished replication with peer 12D3KooWAkfqtGjR23jgLg9YmuDqruMoGJqkaH2bxiRbCnjAPjuQ (1)
[2023-07-24T13:08:50Z INFO  aquadoggo::replication::service] Finished replication with peer 12D3KooWMHjFHCHCQiZDHG5VmTcD4U5SCAyeR9BM82rNBAo7rxWF (4)
[2023-07-24T13:08:51Z INFO  aquadoggo::replication::manager] Accept inbound replication session with peer 12D3KooWKkbEq3owBUBYRCQzruzMskAWt8w2Lqaq7VNGNeSXBXF5 (2)
[2023-07-24T13:08:52Z INFO  aquadoggo::replication::service] Finished replication with peer 12D3KooWKkbEq3owBUBYRCQzruzMskAWt8w2Lqaq7VNGNeSXBXF5 (2)
[2023-07-24T13:08:52Z INFO  aquadoggo::replication::manager] Accept inbound replication session with peer 12D3KooWCHUG9N617CfyMNU8T8XQtS15z7XtJ5NB1MMR3y5WuMC7 (3)
[2023-07-24T13:08:52Z INFO  aquadoggo::replication::service] Finished replication with peer 12D3KooWCHUG9N617CfyMNU8T8XQtS15z7XtJ5NB1MMR3y5WuMC7 (3)
thread 'tokio-runtime-worker' panicked at 'Failed removing completed task from database: Deletion("tasks")', /home/sandreae/Code/aquadoggo/aquadoggo/src/materializer/service.rs:64:26
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
[2023-07-24T13:08:52Z INFO  aquadoggo::manager] Received shutdown signal
@sandreae sandreae added the bug Something isn't working label Jul 24, 2023
@adzialocha
Copy link
Member

Did this come together with other errors or can you observe it independently?

@sandreae
Copy link
Member Author

I didn't see anything else, can check again later 👍

@adzialocha adzialocha mentioned this issue Jul 25, 2023
Closed
@sandreae
Copy link
Member Author 

[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020f0e99aab03e9733098e573ea6635c81541f61be8eaaabf628de0136d18485fe7>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 0020fe5cec0328f8808dddbbbadbc6aa6b0d56e5fca7ce6425de667db05a204cb548
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::worker] Sending materializer reduce task with input <TaskInput -/00202437ce2dfdb0c543057e645b4d30106c1d18b9f62d5cd53e14d846d1838a0875> to the task queue.
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00205b2909a75474969c2b3ae15246ac296e2dc2fc8925aa28ded2ed44f2e763a390
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::reduce] Working on <TaskInput -/00202437ce2dfdb0c543057e645b4d30106c1d18b9f62d5cd53e14d846d1838a0875>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020f0b5b244593cd885c9cdc5b14cd16218f11205331afcc70f069505125d2ca48e>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00205b2909a75474969c2b3ae15246ac296e2dc2fc8925aa28ded2ed44f2e763a390
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00207ea32a34bfcc234d47ca594e4a8105b127aee351232e3bb8d1ecba619290db9d
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020f207f28749682b55dec50ceb4c3fd771f12157b0e796575fa236b0854ec1ca1b>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 00206c675b3bcca23dfbbeb4539f7867c147c02b78fb4584fe0485bcd222632aafca
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 00205b2909a75474969c2b3ae15246ac296e2dc2fc8925aa28ded2ed44f2e763a390
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020f2562fe6c8a84318696c6153e989aaf55c5b2b6b503335da911666fdfa4e27ec>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 00209b7f1cb0b0baae9326512047245fae32c2d3f0085662ce1e0efbd0e3ee0827a6
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00209b7f1cb0b0baae9326512047245fae32c2d3f0085662ce1e0efbd0e3ee0827a6
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020f2e12db41ef5e7d184cf47e08fb4759bafedb57fc2a2900d96d9174b5ec1512f>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 002082baca2468fdd4fbc1978617693c5827021e3921c11bf86e56948655dec748ea
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00206c675b3bcca23dfbbeb4539f7867c147c02b78fb4584fe0485bcd222632aafca
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020f3a7ceb8fb9a534cc16b47bf51d9abf32ddeb11aa70f00095893d16ba7f35e87>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::reduce] No document found for this view, exit without dispatching any other tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00207c192344c305bed08c19dbed227df5418775a65dcbae30a5c8884d347b8ad1c5
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020f43269645e72fec810fe43fa1cef75ad3e5708bfb80431ebc11389c73261c2bb>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00206c675b3bcca23dfbbeb4539f7867c147c02b78fb4584fe0485bcd222632aafca
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 0020e36b0c88960db52fbbd6d169855d7f5a3dccd3b3e90a42d9a7775da7a40257fc
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00205b2909a75474969c2b3ae15246ac296e2dc2fc8925aa28ded2ed44f2e763a390
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020f5862e8b0bfa8fce6004e8fd3fde44ca017796b1dc9b133710ee80d8ae12b6f0>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020f487a8322cf20bdcedc887e6b9c22676677b0de093687079ba5eaa5b6350a6f2>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 00205b2909a75474969c2b3ae15246ac296e2dc2fc8925aa28ded2ed44f2e763a390
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 00207ea32a34bfcc234d47ca594e4a8105b127aee351232e3bb8d1ecba619290db9d
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 0020fe5cec0328f8808dddbbbadbc6aa6b0d56e5fca7ce6425de667db05a204cb548
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020f5b32261c17516c9ba872a3f31f0a8c785f421fdd9f4294f1518ec01132351a0>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00209b7f1cb0b0baae9326512047245fae32c2d3f0085662ce1e0efbd0e3ee0827a6
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020f5a69273a4c2084749f9929fc8136380c90411a647f0a19f0916eb580576ff26>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 00207c192344c305bed08c19dbed227df5418775a65dcbae30a5c8884d347b8ad1c5
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00206c675b3bcca23dfbbeb4539f7867c147c02b78fb4584fe0485bcd222632aafca
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020f66610659ca8064a8bf58fa4b16f7a52a628486154ee52e0c8834dd20684a836>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 0020e708564d836847f98fe07434eb5c02e9d2b202803dce1f180652ebc7373a758a
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 00205b2909a75474969c2b3ae15246ac296e2dc2fc8925aa28ded2ed44f2e763a390
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00205b2909a75474969c2b3ae15246ac296e2dc2fc8925aa28ded2ed44f2e763a390
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020f668aafe7ee973b31a852e4b46aefe982aa9e578b70d1d49d9311e27f8e44384>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 002082baca2468fdd4fbc1978617693c5827021e3921c11bf86e56948655dec748ea
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020f6b2829d95c500a2de14a5f5158f1881b387ef6dc4d26e9aa559c56fddb2c925>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 002082baca2468fdd4fbc1978617693c5827021e3921c11bf86e56948655dec748ea
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 00207c192344c305bed08c19dbed227df5418775a65dcbae30a5c8884d347b8ad1c5
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00207c192344c305bed08c19dbed227df5418775a65dcbae30a5c8884d347b8ad1c5
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020f7044903c0319e308d3b62cc2b7cfc6cf5fa8d7ffde31b9e5b6d5a9de47a61c4>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00207ea32a34bfcc234d47ca594e4a8105b127aee351232e3bb8d1ecba619290db9d
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020f70ca27a6d7f9f69273bd8a0347c13e26a353d333495dd39fa9e295a11dd7e68>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 0020e708564d836847f98fe07434eb5c02e9d2b202803dce1f180652ebc7373a758a
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 00205372950ea8fbf6a22668d292373ad50f13718d53de82573b0523b5ecac135eab
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 0020ee5eb56043e5e1443018c22d58e7cc8e4533b3c9acd1d632734b153fc3ac3874
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 0020e708564d836847f98fe07434eb5c02e9d2b202803dce1f180652ebc7373a758a
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020f7320b0d3d689b0ed7fb109279ed786492ee3c48f331b5d181508d104c6be187>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 0020e36b0c88960db52fbbd6d169855d7f5a3dccd3b3e90a42d9a7775da7a40257fc
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020f75d681d503c049b294f1036ea1453e4e9aa1fefde2fec523a26a66dbec7b339>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 00206c675b3bcca23dfbbeb4539f7867c147c02b78fb4584fe0485bcd222632aafca
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 0020b2ddcfbd3cb439ac6b4656ecf7f89a039de67212cf4dd30d22bbec551af70517
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00205b2909a75474969c2b3ae15246ac296e2dc2fc8925aa28ded2ed44f2e763a390
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020f826ec9af89528a323fd1eab796fb04a33c7bd4e928b4d7f7868a801e506a995>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 00207ea32a34bfcc234d47ca594e4a8105b127aee351232e3bb8d1ecba619290db9d
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 002082baca2468fdd4fbc1978617693c5827021e3921c11bf86e56948655dec748ea
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020f8630d33feacb5e22db7e364e412397b3e47ae5a96ec057856f79f0ec5dcee36>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 0020e36b0c88960db52fbbd6d169855d7f5a3dccd3b3e90a42d9a7775da7a40257fc
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00205b2909a75474969c2b3ae15246ac296e2dc2fc8925aa28ded2ed44f2e763a390
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020f86b31fc0aa5d695c00e1e9b84dcb2e8055ecaf6d74d374f98e953d7a1daa94a>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00207c192344c305bed08c19dbed227df5418775a65dcbae30a5c8884d347b8ad1c5
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020f89c2a5792097dfdbc5ae05df8545b2b75efaf119da2ee4e731410cefc98b855>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 0020e708564d836847f98fe07434eb5c02e9d2b202803dce1f180652ebc7373a758a
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 0020e708564d836847f98fe07434eb5c02e9d2b202803dce1f180652ebc7373a758a
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020f8af107fbe770901c03eef49305e92f35b03e6eaa5dfb0af0bb00fce14ba1966>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 0020b2ddcfbd3cb439ac6b4656ecf7f89a039de67212cf4dd30d22bbec551af70517
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020f8d185f96852e9784a9f5852fff123551b799e9020ac3528953315fd8c2380aa>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 0020ee5eb56043e5e1443018c22d58e7cc8e4533b3c9acd1d632734b153fc3ac3874
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020f91c472f06e23432aefd99c3b23b288c75f65b5fdb05db17c6daa44296a9d984>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00207ea32a34bfcc234d47ca594e4a8105b127aee351232e3bb8d1ecba619290db9d
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00205372950ea8fbf6a22668d292373ad50f13718d53de82573b0523b5ecac135eab
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020f93e53454b3242cdfdbcf7ffbfd8dce8ca0ed46fe4ba7dfd017cbe68ce1239d6>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 0020e708564d836847f98fe07434eb5c02e9d2b202803dce1f180652ebc7373a758a
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 00209b7f1cb0b0baae9326512047245fae32c2d3f0085662ce1e0efbd0e3ee0827a6
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 0020b2ddcfbd3cb439ac6b4656ecf7f89a039de67212cf4dd30d22bbec551af70517
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020f94611cfc96c8ff1e7716f5aa593c581d1784917e1418819a1fce5e456855152>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00206c675b3bcca23dfbbeb4539f7867c147c02b78fb4584fe0485bcd222632aafca
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020f999528efce041b1e88be8d9727dde0438321f0d9ab6892caeee37056986becb>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 0020e708564d836847f98fe07434eb5c02e9d2b202803dce1f180652ebc7373a758a
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020fb0cd5525f31f9d4b6d622de1d8053c7ae8b0a22623461f00cbbad8dd53b53e9>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 0020e36b0c88960db52fbbd6d169855d7f5a3dccd3b3e90a42d9a7775da7a40257fc
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 0020e708564d836847f98fe07434eb5c02e9d2b202803dce1f180652ebc7373a758a
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 00205b2909a75474969c2b3ae15246ac296e2dc2fc8925aa28ded2ed44f2e763a390
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020fb2119735cc53550034c4085352c085669307b8d941f560c622b0eda0b8a84ef>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 0020fe5cec0328f8808dddbbbadbc6aa6b0d56e5fca7ce6425de667db05a204cb548
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 00205b2909a75474969c2b3ae15246ac296e2dc2fc8925aa28ded2ed44f2e763a390
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 0020e708564d836847f98fe07434eb5c02e9d2b202803dce1f180652ebc7373a758a
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 00207ea32a34bfcc234d47ca594e4a8105b127aee351232e3bb8d1ecba619290db9d
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 00207c192344c305bed08c19dbed227df5418775a65dcbae30a5c8884d347b8ad1c5
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020fb30ef886aa4fd5d3bdcf548f06cabec39c54617211ff9b46004b54799ca9bfe>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 0020ee5eb56043e5e1443018c22d58e7cc8e4533b3c9acd1d632734b153fc3ac3874
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 0020b2ddcfbd3cb439ac6b4656ecf7f89a039de67212cf4dd30d22bbec551af70517
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020fb695a954aa7ffd138f66197ed24cbcf8f5252417fe3c7f454e2b82bcbf21957>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00209b7f1cb0b0baae9326512047245fae32c2d3f0085662ce1e0efbd0e3ee0827a6
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020fb834701d711ae582448a6a739d089f20f4f1e0cf56395bf3d8f63ea53f39378>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 0020fe5cec0328f8808dddbbbadbc6aa6b0d56e5fca7ce6425de667db05a204cb548
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020fc0ae2ebd47d146c0036c8d1f8052b1b0021da0cf986b3467cacf63d3d1267f2>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 00205b2909a75474969c2b3ae15246ac296e2dc2fc8925aa28ded2ed44f2e763a390
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00205b2909a75474969c2b3ae15246ac296e2dc2fc8925aa28ded2ed44f2e763a390
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020fce21f7c42d9caa48409d42dc77e2e26c289fdd1ea836b24f3068e7e8e9c3850>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 00205b2909a75474969c2b3ae15246ac296e2dc2fc8925aa28ded2ed44f2e763a390
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 0020e708564d836847f98fe07434eb5c02e9d2b202803dce1f180652ebc7373a758a
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020fd4886e036007b5ab1a905ed303c4e4221eaa9452b12cbcb80ad12c404105735>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 00205b2909a75474969c2b3ae15246ac296e2dc2fc8925aa28ded2ed44f2e763a390
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 00209b7f1cb0b0baae9326512047245fae32c2d3f0085662ce1e0efbd0e3ee0827a6
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00205b2909a75474969c2b3ae15246ac296e2dc2fc8925aa28ded2ed44f2e763a390
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 0020ee5eb56043e5e1443018c22d58e7cc8e4533b3c9acd1d632734b153fc3ac3874
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020fe20f945f633cfa7ebb99774353519390461b3630d6eb42a3c49596c6e6c087e>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020fe2fbba5bc88162b91d7b196675b47edfd9d3ba77c392d1ff5da0dd158aaa808>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00207ea32a34bfcc234d47ca594e4a8105b127aee351232e3bb8d1ecba619290db9d
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020fef1366dc32f1bd5ff291af0ee782aa4fff1eddbcadb3365f881e284c9364bdc>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 002082baca2468fdd4fbc1978617693c5827021e3921c11bf86e56948655dec748ea
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00207c192344c305bed08c19dbed227df5418775a65dcbae30a5c8884d347b8ad1c5
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020ff3cfa3883c18a47b9ad17c6732df49965576bf6a5bd18259af6ca12d55192f4>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 0020e708564d836847f98fe07434eb5c02e9d2b202803dce1f180652ebc7373a758a
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 0020b68c02a0ad430d7f4471af12691c9a477241dbf42d687d237dde196db08ced54
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00205b2909a75474969c2b3ae15246ac296e2dc2fc8925aa28ded2ed44f2e763a390
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 00205b2909a75474969c2b3ae15246ac296e2dc2fc8925aa28ded2ed44f2e763a390
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020ff3d7d9671179c0dd5af51601e39df0767c715272e47357786627f2f0a954965>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00205b2909a75474969c2b3ae15246ac296e2dc2fc8925aa28ded2ed44f2e763a390
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 002082baca2468fdd4fbc1978617693c5827021e3921c11bf86e56948655dec748ea
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020ff78d37d79bf205afaa3b66e0d4eceddf42e8afcd9777044fada2494d34a0005>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020ff865cb65bd272ac7f7d13c0bf56d308305facf0419f3c612ba5b50aaf8a2a31>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 00206c675b3bcca23dfbbeb4539f7867c147c02b78fb4584fe0485bcd222632aafca
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 00206c675b3bcca23dfbbeb4539f7867c147c02b78fb4584fe0485bcd222632aafca
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 0020e36b0c88960db52fbbd6d169855d7f5a3dccd3b3e90a42d9a7775da7a40257fc
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00205b2909a75474969c2b3ae15246ac296e2dc2fc8925aa28ded2ed44f2e763a390
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Working on <TaskInput -/0020ff9b2684efc726932882b892b93f0a757e47073baca762054bd66f9d193fd16e>
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00209b7f1cb0b0baae9326512047245fae32c2d3f0085662ce1e0efbd0e3ee0827a6
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00206c675b3bcca23dfbbeb4539f7867c147c02b78fb4584fe0485bcd222632aafca
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 002082baca2468fdd4fbc1978617693c5827021e3921c11bf86e56948655dec748ea
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 00206c675b3bcca23dfbbeb4539f7867c147c02b78fb4584fe0485bcd222632aafca
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00206c675b3bcca23dfbbeb4539f7867c147c02b78fb4584fe0485bcd222632aafca
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 0020fe5cec0328f8808dddbbbadbc6aa6b0d56e5fca7ce6425de667db05a204cb548
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 00207ea32a34bfcc234d47ca594e4a8105b127aee351232e3bb8d1ecba619290db9d
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 00207ea32a34bfcc234d47ca594e4a8105b127aee351232e3bb8d1ecba619290db9d
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 00207ea32a34bfcc234d47ca594e4a8105b127aee351232e3bb8d1ecba619290db9d
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 002082baca2468fdd4fbc1978617693c5827021e3921c11bf86e56948655dec748ea
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 0020e708564d836847f98fe07434eb5c02e9d2b202803dce1f180652ebc7373a758a
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00205b2909a75474969c2b3ae15246ac296e2dc2fc8925aa28ded2ed44f2e763a390
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 0020e36b0c88960db52fbbd6d169855d7f5a3dccd3b3e90a42d9a7775da7a40257fc
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 0020e708564d836847f98fe07434eb5c02e9d2b202803dce1f180652ebc7373a758a
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00207ea32a34bfcc234d47ca594e4a8105b127aee351232e3bb8d1ecba619290db9d
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 0020b68c02a0ad430d7f4471af12691c9a477241dbf42d687d237dde196db08ced54
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00206c675b3bcca23dfbbeb4539f7867c147c02b78fb4584fe0485bcd222632aafca
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 002082baca2468fdd4fbc1978617693c5827021e3921c11bf86e56948655dec748ea
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00207ea32a34bfcc234d47ca594e4a8105b127aee351232e3bb8d1ecba619290db9d
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Get view for pinned relation with id: 00209b7f1cb0b0baae9326512047245fae32c2d3f0085662ce1e0efbd0e3ee0827a6
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00207ea32a34bfcc234d47ca594e4a8105b127aee351232e3bb8d1ecba619290db9d
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 0020fe5cec0328f8808dddbbbadbc6aa6b0d56e5fca7ce6425de667db05a204cb548
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 0020e708564d836847f98fe07434eb5c02e9d2b202803dce1f180652ebc7373a758a
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 002082baca2468fdd4fbc1978617693c5827021e3921c11bf86e56948655dec748ea
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] View found for pinned relation: 00209b7f1cb0b0baae9326512047245fae32c2d3f0085662ce1e0efbd0e3ee0827a6
[2023-07-25T09:22:23Z DEBUG aquadoggo::materializer::tasks::dependency] Scheduling 0 tasks
[2023-07-25T09:22:23Z INFO  aquadoggo::replication::service] Closed connection with peer: 12D3KooWQ2f9NVHaUF1mqbADtA3L7w1qgyB3xJCp5RxQvZSRRTHT (7)
[2023-07-25T09:22:23Z INFO  aquadoggo::replication::service] Closed connection with peer: 12D3KooWQ2f9NVHaUF1mqbADtA3L7w1qgyB3xJCp5RxQvZSRRTHT (2)
[2023-07-25T09:22:23Z INFO  aquadoggo::replication::service] Closed connection with peer: 12D3KooWPxN5uuvcR8HWQMz4oR1N2iQ9VhX2LNQzvzKoZnsEzoie (1)
[2023-07-25T09:22:23Z INFO  aquadoggo::replication::service] Closed connection with peer: 12D3KooWPxN5uuvcR8HWQMz4oR1N2iQ9VhX2LNQzvzKoZnsEzoie (9)
[2023-07-25T09:22:23Z INFO  aquadoggo::replication::service] Closed connection with peer: 12D3KooW9waB24G8TfQcZL5oKmoiSaADvPNF72GD5ei3jgqf94HH (1)
[2023-07-25T09:22:23Z INFO  aquadoggo::replication::service] Closed connection with peer: 12D3KooW9waB24G8TfQcZL5oKmoiSaADvPNF72GD5ei3jgqf94HH (1)
[2023-07-25T09:22:23Z INFO  aquadoggo::replication::service] Closed connection with peer: 12D3KooWMHjFHCHCQiZDHG5VmTcD4U5SCAyeR9BM82rNBAo7rxWF (1)
thread 'tokio-runtime-worker' panicked at 'Failed removing completed task from database: Deletion("tasks")', /home/sandreae/Code/aquadoggo/aquadoggo/src/materializer/service.rs:64:26
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
[2023-07-25T09:22:44Z INFO  aquadoggo::manager] Received shutdown signal

@sandreae
Copy link
Member Author

After restarting the node:

After restarting the same node (where the above logs came from) I then get this error which I haven't seen before:

[2023-07-25T09:25:30Z WARN  aquadoggo::replication::manager] Tried to remove nonexistent session 0 with peer: 12D3KooWQ2f9NVHaUF1mqbADtA3L7w1qgyB3xJCp5RxQvZSRRTHT (1)
[2023-07-25T09:25:30Z INFO  aquadoggo::replication::manager] Initiate outbound replication session with peer 12D3KooW9waB24G8TfQcZL5oKmoiSaADvPNF72GD5ei3jgqf94HH (1)
[2023-07-25T09:25:30Z INFO  aquadoggo::replication::manager] Initiate outbound replication session with peer 12D3KooWQ2f9NVHaUF1mqbADtA3L7w1qgyB3xJCp5RxQvZSRRTHT (1)
[2023-07-25T09:25:30Z WARN  aquadoggo::replication::service] Replication with peer 12D3KooW9waB24G8TfQcZL5oKmoiSaADvPNF72GD5ei3jgqf94HH (1) failed: Incoming data could not be ingested: Error occured during `LogStorage` request in storage provider: error returned from database: (code: 2067) UNIQUE constraint failed: logs.public_key, logs.log_id
[2023-07-25T09:25:30Z WARN  aquadoggo::replication::service] Replication with peer 12D3KooW9waB24G8TfQcZL5oKmoiSaADvPNF72GD5ei3jgqf94HH (1) failed: No session found with id 0 for peer 12D3KooW9waB24G8TfQcZL5oKmoiSaADvPNF72GD5ei3jgqf94HH (1)
[2023-07-25T09:25:30Z WARN  aquadoggo::replication::manager] Tried to remove nonexistent session 0 with peer: 12D3KooW9waB24G8TfQcZL5oKmoiSaADvPNF72GD5ei3jgqf94HH (1)
[2023-07-25T09:25:30Z WARN  aquadoggo::replication::service] Replication with peer 12D3KooWQ2f9NVHaUF1mqbADtA3L7w1qgyB3xJCp5RxQvZSRRTHT (1) failed: Incoming data could not be ingested: Error occured during `LogStorage` request in storage provider: error returned from database: (code: 2067) UNIQUE constraint failed: logs.public_key, logs.log_id
[2023-07-25T09:25:30Z WARN  aquadoggo::replication::service] Replication with peer 12D3KooWQ2f9NVHaUF1mqbADtA3L7w1qgyB3xJCp5RxQvZSRRTHT (1) failed: No session found with id 0 for peer 12D3KooWQ2f9NVHaUF1mqbADtA3L7w1qgyB3xJCp5RxQvZSRRTHT (1)
[2023-07-25T09:25:30Z WARN  aquadoggo::replication::manager] Tried to remove nonexistent session 0 with peer: 12D3KooWQ2f9NVHaUF1mqbADtA3L7w1qgyB3xJCp5RxQvZSRRTHT (1)
^C[2023-07-25T09:25:31Z INFO  aquadoggo::manager] Received shutdown signal

@sandreae
Copy link
Member Author

sandreae commented Jul 25, 2023 •
edited
Loading

aquadoggo/aquadoggo/src/db/stores/task.rs

Lines 88 to 92 in 95a8776

if result.rows_affected() != 1 {
Err(SqlStoreError::Deletion("tasks".into()))
} else {
Ok(())
}

Removing this condition (that exactly one row is affected) means the issue goes away. We need to check this is the intended behaviour though.

@adzialocha
Copy link
Member

Did you have a chance to log what the value is?

@sandreae
Copy link
Member Author

Ah no, I'll do that now.

@sandreae
Copy link
Member Author 

1cb>
[2023-07-25T15:30:56Z INFO  aquadoggo::materializer::tasks::reduce] Created <Document 6b1f27>
[2023-07-25T15:30:56Z INFO  aquadoggo::materializer::tasks::reduce] Created <Document 8db752>
[2023-07-25T15:30:56Z INFO  aquadoggo::materializer::tasks::reduce] Created <Document 5d8b48>
[2023-07-25T15:30:57Z INFO  aquadoggo::materializer::tasks::reduce] Created <Document 895146>
Rows affected: 0
thread 'tokio-runtime-worker' panicked at 'Failed removing completed task from database: Deletion("tasks")', /home/sandreae/Code/aquadoggo/aquadoggo/src/materializer/service.rs:64:26
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
[2023-07-25T15:30:57Z INFO  aquadoggo::manager] Received shutdown signal
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3450 w. <TaskInput -/00207cdcf9062aea2d71d30d0af0b4ec325d8897fa9ffff983d8b07d9d357e9fb3b2>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3451 w. <TaskInput -/00207df669ab2a382d0cbf3fdae6ac5c72287ce9d8dcbd719945129789389c063db2>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3440 w. <TaskInput -/0020773f6ab92a2cdd65bd387cf6e2a7094991b864fb6203042b64ffec66904725dd>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3452 w. <TaskInput -/00207f18183ee6ba969c7059d4def7d68f501bcf7d8debd1d097fc937662f834e0ad>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3454 w. <TaskInput -/00207cac023136e2e5a72fe8aadc505b8661f5e0548df8d635dc3454a1e4b6846817>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3445 w. <TaskInput -/00207c28219d9f9705ac489d983a1fc29e10d07a2e2f152982e66376688c4b84f898>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3455 w. <TaskInput -/00207fa1f5e1dac0bb26c62779ca2b45cdcaabad6815e22d71e39116b2bbf717e91f>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3457 w. <TaskInput -/00207fd8b6007bae8be1edc54f94e626294ab1ac4f6abb08825e80ddfe27a156d1c7>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3456 w. <TaskInput -/00207fc7df7daad3ab8dd7533a17f6cbd828d015c8c417eab0d9fb77a2e5dc6fd94c>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3458 w. <TaskInput -/0020825a17414858b7ee0226e728c3a8db1a88b253a1c3b7ba5e72211b484dfc2ee9>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3459 w. <TaskInput -/0020807055415a9312b5763159442fa75f84858e1993d58439f26a734d1ebf4f9039>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3460 w. <TaskInput -/0020812282ec94e36689c15310afc0e0b461abbefcbea54f6169a1b3fcf433a0cbee>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3461 w. <TaskInput -/00208abae54f355f17291effffedd895a8ca21bfb280ecfc4eddceb695dfb33d11cb>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3462 w. <TaskInput -/00208364c10c4bfa6d62d14d14fbf2bba2d3f1c3fbeb67e375c5e047c886a7c6004e>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3464 w. <TaskInput -/002080d2360fc0ddc1dce3ec10cdac99af3151af3fc04f0082a0f06c521810d29984>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3465 w. <TaskInput -/00207cf2b9b6735fc9b33ecd64ed59624fab7d71a2147de91bdb60d953854d87b6be>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3463 w. <TaskInput -/002082e737072fb16e3fbaf8ffd15833e6119316affad2d79842eeecb2c1bfddf499>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3442 w. <TaskInput -/002076c13be295278cfc0b19adf874a1c2575f4c67b4cc2c42e2c0dae32e3f8d66d3>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3466 w. <TaskInput -/002084c24827e235ca19bb80f0b2f86c5ce03f138ac32f2e62b405bba1411a6ce670>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3467 w. <TaskInput -/00207ffba2fa50ac2925a4fcdecd681e560d3b1cf892c8fa31a9340fd1d434d92af3>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3468 w. <TaskInput -/002083b9eca2a5b9cb743059645ac8afacc9f8838ab2a211f88bcb94dd2fdb34f1d2>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3469 w. <TaskInput -/0020852d1f845d2d44f13525af477d2324978e29a1e1802aa31288ab6b9f2b153204>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3470 w. <TaskInput -/00208ac091d0190e2f47a782dbe315c4e023d66991e36d308e5cc5f8cf00706b1f27>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3471 w. <TaskInput -/00208aecb298dbece513954eb8f833ad3a13e57f746b1208b4aa77ca9d62458db752>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3472 w. <TaskInput -/00208af242f43ed34adfa49ef896f0962a2a05fa8139640e970de39539b68f5d8b48>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3473 w. <TaskInput -/00208b1b9379b59c773d375fac50ef8cd02098c2ff78f67d603497d979170c895146>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3446 w. <TaskInput -/00207bb2ca2bd6a08891b9eea62324a6a3cc83c99343471f99b469415d09a0cdb091>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3443 w. <TaskInput -/00207d139902afded4c9796ae23b36a15348d781d67425b5d83a2cca16757c6cddcb>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3449 w. <TaskInput -/002079df6897dda59548ec9ad56a95e2f05da7f049406e126e8bd0096ad1b18cc31d>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3453 w. <TaskInput -/002079f32595a41dae032cb4e43433cc4af64c91dc01a59f5ce8a9c1e09dab055254>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3447 w. <TaskInput -/00207f37774e9d4bfc4fd6f0c96f23eee1fbc85a15bf1c20c5969dd771aab070e92e>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3438 w. <TaskInput -/00207aa9fd46f7d85d45805c85c239ac55b4afde919ec9e129c9d88d5ac7ddc3758f>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3441 w. <TaskInput -/0020794255b67ccdfb9a8a3a3ad773585e6d7bb82511932e81c8f74719582c6ae609>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker reduce with task <QueueItem 2348 w. <TaskInput -/0020b2ddcfbd3cb439ac6b4656ecf7f89a039de67212cf4dd30d22bbec551af70517>>: A fatal error occured in OperationStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker reduce with task <QueueItem 2347 w. <TaskInput -/0020b68c02a0ad430d7f4471af12691c9a477241dbf42d687d237dde196db08ced54>>: A fatal error occured in OperationStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3444 w. <TaskInput -/00207c0a08b90a5d6815dd965832c2435cab59442d9991c96ff883dba85ca8ef2155>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3448 w. <TaskInput -/00207e5f9551291e2d30da1f7f47ef896b7844f285421072396c150d59f937d1acb7>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3439 w. <TaskInput -/0020757d703947327a1370d883bc428076233896c7219b1ff3f68c62ca33ac5b9087>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3436 w. <TaskInput -/00207adda78d85275127b5bf4adbb4b08c318d31784db24df5ccb29e9c197b6aec59>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3437 w. <TaskInput -/0020758d86d33f3f6a03ae329ed6f767ee64988884802b5cb1d341822efc31f5b151>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool
[2023-07-25T15:30:57Z INFO  aquadoggo::materializer::tasks::reduce] Created <Document f944aa>
[2023-07-25T15:30:57Z ERROR aquadoggo::materializer::worker] Critical error in worker dependency with task <QueueItem 3474 w. <TaskInput -/00208b8aade618f7460ebc05d9b0ff5173312ef8e17c6a1a972732dc4d436af944aa>>: A fatal error occured in DocumentStore: attempted to acquire a connection on a closed pool

@sandreae
Copy link
Member Author

Rows affected: 0

@sandreae sandreae mentioned this issue Jul 25, 2023
Merged
4 tasks
@sandreae
Copy link
Member Author

#461

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Don't check affected rows when deleting task p2panda/aquadoggo
2 participants
@adzialocha @sandreae