Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Renaming obfuscated files during downloading #362

Closed
hugbug opened this issue Apr 16, 2017 · 0 comments
Closed

Renaming obfuscated files during downloading #362

hugbug opened this issue Apr 16, 2017 · 0 comments
Labels
Milestone

Comments

@hugbug
Copy link
Member

@hugbug hugbug commented Apr 16, 2017

Many files are posted to Usenet with obfuscated names. The original file names are needed to successfully unpack downloaded files. NZBGet has built-in automatic renaming. Renaming (par-renaming and rar-renaming) is a post-processing stage where original file names are restored using data from par2-files or meta-information from rar-files. Currently downloading and renaming are separated, which makes the code simpler and more maintainable.

This issue is to investigate and experiment with built-in automatic file renaming during downloading. This feature is a prerequisite for unpack during downloading which is the ultimate goal.

Steps to rename obfuscated files as they are downloaded:

  • download first article of each file included in nzb;
  • analyse first kilobytes of each file to determine which of the file is a par2-file;
  • using first articles calculate 16k hashes for each downloaded file;
  • download one par2-file completely;
  • load par2-file and read 16k hashes;
  • rename already completed files if there are any;
  • set correct names for incomplete files;
  • let download continue with correct names.

Advanced version also needs:

  • support multiple par-sets:
    • detect par-sets;
    • download one par2-file for each par-sets;
    • use every loaded par2-files for renaming;
  • rename par2-files too;
  • properly handle connection interruptions;
  • handle partially downloaded (unnecessary) par2-files (.out.tmp-files):
    • do not leave .out.tmp-files for successful downloads;
    • par-rename and par-check should play nice with partially downloaded files (.out.tmp-files);
    • force cache flushing for partially downloaded files before post-processing;
  • save and restore renaming state if the program was reloaded before renaming completed;
  • new event for queue-scripts; to send after renaming.
@hugbug hugbug added the feature label Apr 16, 2017
@hugbug hugbug added this to the v19 milestone Apr 16, 2017
hugbug added a commit that referenced this issue Apr 22, 2017
hugbug added a commit that referenced this issue Apr 22, 2017
, then other articles, when option “DirectRename” is active.
hugbug added a commit that referenced this issue Apr 22, 2017
when direct write was disabled
hugbug added a commit that referenced this issue Apr 22, 2017
hugbug added a commit that referenced this issue Apr 22, 2017
(new tests currently fail)
hugbug added a commit that referenced this issue Apr 22, 2017
hugbug added a commit that referenced this issue Apr 22, 2017
hugbug added a commit that referenced this issue Apr 23, 2017
as we need it to download first and ExtraPriority-flag doesn’t work on
paused files
hugbug added a commit that referenced this issue Apr 23, 2017
hugbug added a commit that referenced this issue Apr 24, 2017
hugbug added a commit that referenced this issue Apr 24, 2017
if they happen during downloading of first articles
hugbug added a commit that referenced this issue Apr 25, 2017
hugbug added a commit that referenced this issue Apr 28, 2017
Fixed: if the file was renamed during finalizing stage (completion) the
file may not be properly renamed.
hugbug added a commit that referenced this issue Apr 28, 2017
to solve issues with par-checker, par-renamer and avoid saving state
for these files.
hugbug added a commit that referenced this issue Apr 28, 2017
Spread RenameInfo into FileInfo and NzbInfo to make the state easier to
save on disk
hugbug added a commit that referenced this issue Apr 29, 2017
hugbug added a commit that referenced this issue May 1, 2017
Queue-event NZB_NAMED, sent after the inner files are renamed
@hugbug hugbug closed this May 1, 2017
@hugbug hugbug mentioned this issue May 3, 2017
8 of 8 tasks complete
hugbug added a commit that referenced this issue May 16, 2017
hugbug added a commit that referenced this issue May 23, 2017
If a file got lost before cache flushing.
hugbug added a commit that referenced this issue May 23, 2017
Now also discarding data when download completes without direct rename
being able to process files (due to download errors).
hugbug added a commit that referenced this issue Oct 9, 2017
, then other articles, when option “DirectRename” is active.
hugbug added a commit that referenced this issue Oct 9, 2017
when direct write was disabled
hugbug added a commit that referenced this issue Oct 9, 2017
hugbug added a commit that referenced this issue Oct 9, 2017
(new tests currently fail)
hugbug added a commit that referenced this issue Oct 9, 2017
hugbug added a commit that referenced this issue Oct 9, 2017
hugbug added a commit that referenced this issue Oct 9, 2017
as we need it to download first and ExtraPriority-flag doesn’t work on
paused files
hugbug added a commit that referenced this issue Oct 9, 2017
hugbug added a commit that referenced this issue Oct 9, 2017
if they happen during downloading of first articles
hugbug added a commit that referenced this issue Oct 9, 2017
hugbug added a commit that referenced this issue Oct 9, 2017
Fixed: if the file was renamed during finalizing stage (completion) the
file may not be properly renamed.
hugbug added a commit that referenced this issue Oct 9, 2017
to solve issues with par-checker, par-renamer and avoid saving state
for these files.
hugbug added a commit that referenced this issue Oct 9, 2017
Spread RenameInfo into FileInfo and NzbInfo to make the state easier to
save on disk
hugbug added a commit that referenced this issue Oct 9, 2017
Queue-event NZB_NAMED, sent after the inner files are renamed
hugbug added a commit that referenced this issue Oct 9, 2017
hugbug added a commit that referenced this issue Oct 9, 2017
If a file got lost before cache flushing.
hugbug added a commit that referenced this issue Oct 9, 2017
Now also discarding data when download completes without direct rename
being able to process files (due to download errors).
hugbug added a commit that referenced this issue Oct 9, 2017
which may happen if download errors occurred.
hugbug added a commit that referenced this issue Oct 9, 2017
which may happen if articles were yEncoded with less than 68 character
length.
hugbug added a commit that referenced this issue Oct 9, 2017
hugbug added a commit that referenced this issue Oct 9, 2017
are now correctly processed by direct rename.
hugbug added a commit that referenced this issue Oct 9, 2017
hugbug added a commit that referenced this issue Jan 17, 2019
That’s needed to make direct rename work properly if the program was
reloaded in the middle of direct rename download process.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.