error persisting torrents: extended protocol limited to 65535 parameters #126
Comments
|
Hi @davispuh , I suspect it's the Any reason for enabling this setting? Aside from any other issue it will use a LOT more disk space and the pieces can't actually be used for anything right now. We might do something with the pieces in future but it will probably involve saving pieces for a few select torrents rather than everything.... It would be helpful to check if disabling this resolves the issue, then at least we'll know for sure if it's that. |
Yeah I have my own use case for that, I want to collect all information and storage is not issue. It looks like we're not saving nodes either - would want those aswell.
That looks like just one of the ways you can trigger this, but I think torrent with more than 65k files could also trigger this aswell along with other combinations. The root issue seems that when you See this https://klotzandrew.com/blog/postgres-passing-65535-parameter-limit |
|
I also run into this problem with The following logs were trying to persisting 18 torrents with a total of 26693 files.{
severity: 'ERROR',
timestamp: '2024-04-08T13:30:32.067571475Z',
logger: 'gorm',
caller: 'logger/logger.go:72',
message: 'gorm trace',
location: '/build/internal/database/dao/torrents.gen.go:766',
error: 'extended protocol limited to 65535 parameters',
elapsed: 264.841844,
sql: 'INSERT INTO "torrent_files" ("info_hash","index","path","size","created_at","updated_at") VALUES (................26693 rows................) ON CONFLICT DO NOTHING',
rows: 0,
stacktrace: 'github.com/bitmagnet-io/bitmagnet/internal/database/logger.(*customLogger).Trace\n' +
'\t/build/internal/database/logger/logger.go:72\n' +
'gorm.io/gorm.(*processor).Execute\n' +
'\t/go/pkg/mod/gorm.io/gorm@v1.25.5/callbacks.go:134\n' +
'gorm.io/gorm.(*DB).Create\n' +
'\t/go/pkg/mod/gorm.io/gorm@v1.25.5/finisher_api.go:24\n' +
'gorm.io/gorm/callbacks.saveAssociations\n' +
'\t/go/pkg/mod/gorm.io/gorm@v1.25.5/callbacks/associations.go:431\n' +
'gorm.io/gorm/callbacks.RegisterDefaultCallbacks.SaveAfterAssociations.func4\n' +
'\t/go/pkg/mod/gorm.io/gorm@v1.25.5/callbacks/associations.go:258\n' +
'gorm.io/gorm.(*processor).Execute\n' +
'\t/go/pkg/mod/gorm.io/gorm@v1.25.5/callbacks.go:130\n' +
'gorm.io/gorm.(*DB).CreateInBatches.func1\n' +
'\t/go/pkg/mod/gorm.io/gorm@v1.25.5/finisher_api.go:48\n' +
'gorm.io/gorm.(*DB).Transaction\n' +
'\t/go/pkg/mod/gorm.io/gorm@v1.25.5/finisher_api.go:633\n' +
'gorm.io/gorm.(*DB).CreateInBatches\n' +
'\t/go/pkg/mod/gorm.io/gorm@v1.25.5/finisher_api.go:60\n' +
'gorm.io/gen.(*DO).CreateInBatches\n' +
'\t/go/pkg/mod/gorm.io/gen@v0.3.25/do.go:598\n' +
'github.com/bitmagnet-io/bitmagnet/internal/database/dao.torrentDo.CreateInBatches\n' +
'\t/build/internal/database/dao/torrents.gen.go:766\n' +
'github.com/bitmagnet-io/bitmagnet/internal/dhtcrawler.(*crawler).runPersistTorrents.func2\n' +
'\t/build/internal/dhtcrawler/persist.go:66\n' +
'github.com/bitmagnet-io/bitmagnet/internal/dhtcrawler.(*crawler).runPersistTorrents.(*Query).Transaction.func3\n' +
'\t/build/internal/database/dao/gen.go:196\n' +
'gorm.io/gorm.(*DB).Transaction\n' +
'\t/go/pkg/mod/gorm.io/gorm@v1.25.5/finisher_api.go:647\n' +
'github.com/bitmagnet-io/bitmagnet/internal/database/dao.(*Query).Transaction\n' +
'\t/build/internal/database/dao/gen.go:196\n' +
'github.com/bitmagnet-io/bitmagnet/internal/dhtcrawler.(*crawler).runPersistTorrents\n' +
'\t/build/internal/dhtcrawler/persist.go:57'
}{
severity: 'ERROR',
timestamp: '2024-04-08T13:30:32.110365228Z',
logger: 'gorm',
caller: 'logger/logger.go:72',
message: 'gorm trace',
location: '/build/internal/database/dao/torrents.gen.go:766',
error: 'extended protocol limited to 65535 parameters',
elapsed: 660.740443,
sql: 'INSERT INTO "torrents" ("info_hash","name","size","private","created_at","updated_at","files_status","files_count") VALUES (................about 18 items................) ON CONFLICT ("info_hash") DO UPDATE SET "name"="excluded"."name","files_status"="excluded"."files_status","files_count"="excluded"."files_count","updated_at"="excluded"."updated_at"',
rows: 20,
stacktrace: 'github.com/bitmagnet-io/bitmagnet/internal/database/logger.(*customLogger).Trace\n' +
'\t/build/internal/database/logger/logger.go:72\n' +
'gorm.io/gorm.(*processor).Execute\n' +
'\t/go/pkg/mod/gorm.io/gorm@v1.25.5/callbacks.go:134\n' +
'gorm.io/gorm.(*DB).CreateInBatches.func1\n' +
'\t/go/pkg/mod/gorm.io/gorm@v1.25.5/finisher_api.go:48\n' +
'gorm.io/gorm.(*DB).Transaction\n' +
'\t/go/pkg/mod/gorm.io/gorm@v1.25.5/finisher_api.go:633\n' +
'gorm.io/gorm.(*DB).CreateInBatches\n' +
'\t/go/pkg/mod/gorm.io/gorm@v1.25.5/finisher_api.go:60\n' +
'gorm.io/gen.(*DO).CreateInBatches\n' +
'\t/go/pkg/mod/gorm.io/gen@v0.3.25/do.go:598\n' +
'github.com/bitmagnet-io/bitmagnet/internal/database/dao.torrentDo.CreateInBatches\n' +
'\t/build/internal/database/dao/torrents.gen.go:766\n' +
'github.com/bitmagnet-io/bitmagnet/internal/dhtcrawler.(*crawler).runPersistTorrents.func2\n' +
'\t/build/internal/dhtcrawler/persist.go:66\n' +
'github.com/bitmagnet-io/bitmagnet/internal/dhtcrawler.(*crawler).runPersistTorrents.(*Query).Transaction.func3\n' +
'\t/build/internal/database/dao/gen.go:196\n' +
'gorm.io/gorm.(*DB).Transaction\n' +
'\t/go/pkg/mod/gorm.io/gorm@v1.25.5/finisher_api.go:647\n' +
'github.com/bitmagnet-io/bitmagnet/internal/database/dao.(*Query).Transaction\n' +
'\t/build/internal/database/dao/gen.go:196\n' +
'github.com/bitmagnet-io/bitmagnet/internal/dhtcrawler.(*crawler).runPersistTorrents\n' +
'\t/build/internal/dhtcrawler/persist.go:57'
}{
severity: 'ERROR',
timestamp: '2024-04-08T13:30:32.112227619Z',
logger: 'dht_crawler',
caller: 'dhtcrawler/persist.go:74',
message: 'error persisting torrents: extended protocol limited to 65535 parameters',
stacktrace: 'github.com/bitmagnet-io/bitmagnet/internal/dhtcrawler.(*crawler).runPersistTorrents\n' +
'\t/build/internal/dhtcrawler/persist.go:74'
} |
|
I tried to workaround it with diff --git a/internal/dhtcrawler/persist.go b/internal/dhtcrawler/persist.go
index f144234..c53fe92 100644
--- a/internal/dhtcrawler/persist.go
+++ b/internal/dhtcrawler/persist.go
@@ -63,7 +63,7 @@ func (c *crawler) runPersistTorrents(ctx context.Context) {
string(c.dao.Torrent.FilesCount.ColumnName()),
string(c.dao.Torrent.UpdatedAt.ColumnName()),
}),
- }).CreateInBatches(ts, 20); err != nil {
+ }).CreateInBatches(ts, 1); err != nil {
return err
}
if err := tx.WithContext(ctx).QueueJob.CreateInBatches(jobs, 10); err != nil {But that didn't got rid of it, still happens sometimes. |
|
Yeah, I think the batch affects only the |
|
I looked into this and for me torrent with most files that have been successfully saved contains 14147 files. Each I also looked into torrents that failed to save and I have thousands of them with more than 14k files. For example largest torrent I see contains 48460 files (it's ~1.5 TB torrent lmaoo and even actively seeded). So basically currently we are losing all torrents with ~14k+ files. Solution would be to split it up in multiple smaller queries and don't try to submit it all in one go. PS. qBittorrent handled that big torrent perfectly fine without issues |
Have you checked the roadmap on the website, and existing issues, before opening a dupllcate issue?
Yes
Describe the bug
In logs can see PostgreSQL error:
To Reproduce
Not sure, but maybe due to
save_pieces: true.Expected behavior
No errors :)
General (please complete the following information):
The text was updated successfully, but these errors were encountered: