parity files from lower level servers will likely cheaper than articles from higher level servers

[nomandera]

As a a test I recently put some effort into allocating higher level servers that utilise different upsteam peering. One of these test "fills" is a comparatively expensive block account.

As you would expect overall article completion has come close to 100% but I noticed I was consuming more block credit than I had predicted.

What I believe is happening is that nzbget is doing its job and finding as many articles as it can. Given the peering in this test, parity recovery files are seldom being used as most articles can be sourced from usenet.

The issue is that the low level servers whilst incomplete in my case are fixed price for unlimited usage and therefore free in this context, whilst the block account is actually quite expensive. At some point the completion percentage will be such that parity files from the "free" lower level server could be used to complete the posting rather than consume expensive block credit.

Thoughts?

[hugbug]

Yes but who wants to wait days until repair completes?

[nomandera]

Yes that would be valid argument with large sets but for the overnight multiple separate 1GB file sets the repair time is so short as to always be complete in the morning. These are the ones that will consume the "hidden" big numbers over a year.

[hugbug]

The feature could be useful but it's hard to find a balance between repair time and block server usage.

I've made a test with a 1GB file where I removed three files each 50MB to simulate the use case. The repair took 2 hours (slow CPU comparable with older NAS). Is it long or not? I guess the answer is "it depends". If it's happening at night and there are no other downloads queued - that's OK. But if there are other downloads of popular shows waiting for the first, it could happen that the wait time makes the things worse for them as they may start disappearing from server.

The things become even more complicated when we take nzbs of different sizes into account: up to 40 GB. The whole system will need a complex configuration to tell NZBGet which strategy to choose when.

I do like the current strategy "download things as fast as possible from all sources". As a developer I aim to keep things simpler. However if there is more demand for such a feature and we can find an easy way for user to configure it without 20 new options - this is something that may happen.

I don't know how to involve more users into discussion as the issue tracker isn't much popular for feature talks. May be a cross-post on forum in feature discussion section?

[RollingStar]

I agree with OP. Just like @anoma describes, I find myself using too much block quota because I bought a block account. My block can get used up quickly because parity files aren't being downloaded. NZBget could prioritize all available articles on "free" / "unlimited" / "high priority" servers.

I'm doing intensive PAR repairs on two machines:

• gaming PC with 4-core AMD Phenom II X4 965 @ 3.4 GHz
• few-years-old Dell with an Intel Core 2 Duo @ 3.0 GHz

I can repair 50 GB downloads that have <10% damage in around 50 minutes on the gaming PC.

I recognize that old hardware or a Raspberry Pi are not going to do PAR repairs quickly. But I would love to have this feature from NZBget.

The end user will pay one way or another. Either they will pay for additional Usenet servers to improve file availability, or they will pay for hardware and electricity for PAR repairs.

[deanishe]

and we can find an easy way for user to configure it without 20 new options

How about a per-server "is block account" toggle and a global "Prefer parity files to block accounts" toggle? Perhaps with an optional threshold priority above which the PAR setting is ignored and files are downloaded as fast as possible, like they currently are?

[Panja0]

I would like to chip in as well.

What I'll like to see is that when my main server is getting incomplete download and the health reaches a certain %, for instance 95%, that is the sign for the block/fill server to come in to play.
But only after a certain % of failure/health. This will save lots of GB's on my block account.

[rovingrambler]

I like deanishe's idea. You could have it off on a cheap block, more selective with an expensive one.
IDN that a seperate par toggle is needed, I think just heavier weighted toward unlimited/inexpensive that would imply "prefer parity" prior to fill.
That plus select-able critical health value. flaking out at 90% would save me TBs a year.

Fantastic software hugbug!

[trafgals]

One year later - any way to do this?