Feature suggestion: parallel rar/par and upload queue

[Tensai75]

Hi Matthieu

Have you ever thought of separating the rar/par process and and the upload process into separate queues so they can be done in parallel?
Especially when monitoring a folder with a lot of jobs, this could drastically improve the overall performance. E.g. while uploading the first job (after it has been initially rared and pared), the next job(s) could already be rared and pared in parallel and then put into the upload queue.
Programmatically this can be done easily and I used this approach in a private upload tool I wrote myself in node.js using nyuu as the actual upload program.

Would love to hear your thoughts about this.

Regards, Tensai

[mbruel]

Hi Tensai,
You can do obviously do it... It won't be that easy within ngPost but not the end of the world either.
There are pros and cons, personally I see more cons.

1.: pros:

• potentially you never stop uploading once the first rar/par2 is done until you've consumed all your queue.
  The upload rate doesn't improve BUT the overall time for your post could decrease significantly.
  I've never touched 4k releases around 50GB, but it seems to me that packing (rar -m0) and generating par2 (with ParPar) is quite fast, a detail compare to the upload duration, even if you've a 10GBps connection no?

2.: cons:

• you use more disk space as you have 2 or even several temporary compression folders. Imagine your queue has only 50GB folders, if the packing is way faster and you end up with 10 temporary folders. That's 500GB! You may run out of disk.
• ParPar can be CPU consuming. How do I make sure that the Threads are well spread between ngPost and ParPar? i.e: you shouldn't exceed your number of core, so limit both ngPost and ParPar.

So basically I find it too risky. And anyway, if you're having a queue, what's the hurry, just launch it and come back the next morning.

[Tensai75]

Hi Matthieu

many thanks for our response.
Personally I think the pros upweight the cons by far, especially for "professional" uploaders which have separate scripts to continuously fill the monitored folder with new jobs and where it is key to have all the files uploaded as fast as possible.
Disk space usually is no issue for them. But to be on the safe side, a mechanism could be implemented to simply pause the rar/par queue if disk space is low. As soon as uploaded files are deleted again and enough disk space is free, the rar/par queue could resume again.
And as you mentioned, limiting factor is almost always the upload itself. So there is no need to have rar/par working at the disk i/o limit (which usually is the limit for rar/par) and they could simply be limited to one thread each.

Unfortunately I am not as proficiency in C++/QT as you obviously are and honestly have no time to dig into it.
But be assured that there is a huge interest in this option and ngPost could become the standard usenet upload tool if this would be implemented.

Still hope you might consider doing so.

Regards, Tensai

[mbruel]

Well I didn't have any feedback from "professional" posters...
I'm not sure if any use ngPost. I suppose they already have their home-made scripts and continue to use them, which makes sense. Probably they still use Nyuu as they don't really have any reason to change...

Monitor the disk space is an hassle, especially with multi-OS support.
I could maybe implement to only have the next Job packed in advance, which means no more than 2 temporary folders, so I imagine, max 200GB.

I'm waiting for some translations to release the v4.8. I'll check if I can find a way to include it "easily".

[Tensai75]

Well I didn't have any feedback from "professional" posters...

Well, now you have ;-) At least indirect because I am not actually one of them myself but I certainly can say I do represent some of them.
And I actually tried to convince them to switch to ngPost because I have no time to maintain my tool any further. And guess what, main reason why they didn't want to was the missing parallel rar/par and upload process.
But what they did like very much was the multi server support and also the UI. Two things that nyuu does not offer on its own. And of course also the speed which is beyond nyuu.
So as said, your program has already been very positively noticed and its potential is huge.

I could maybe implement to only have the next Job packed in advance, which means no more than 2 temporary folders, so I imagine, max 200GB.

This is absolutely reasonable and would be sufficient to achive the same overall performance.
Would be great to have this!

[mbruel]

ok I'm having a look.
could you drop me an email, maybe you could help for the German translation of 6 sentences for the new release, I'll add you to my mailing list. (Matthieu.Bruel@gmail.com)

[mbruel]

@Mamitomama I'm not sure I get what you want to do... Why stop the upload and/or only prepare the packing without uploading? what's the goal.

I've implemented a simple version where I'm packing the next post in queue while uploading the current one. It seems to work. I believe it is enough as packing is in general much faster than posting. So you end up waiting only the first time for the packing then it is uploading in continue the whole queue \o/

Would you guys be able to beta test? I'm still missing a little part to handle new incoming file but it should be ready in a few hours.

What OS are you on? Linux command line? do you compile from the sources?

PS: @Mamitomama are you on the same community of @Tensai75 ?

[mbruel]

the beta is pushed.
Please let me know your feedback.

Open the config file to add PREPARE_PACKING:

## when several Posts are queued, prepare the packing of the next Post while uploading the current one
PREPARE_PACKING = true