New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Renaming obfuscated files during downloading #362

Closed
hugbug opened this Issue Apr 16, 2017 · 0 comments

Comments

Projects
None yet
1 participant
@hugbug
Member

hugbug commented Apr 16, 2017

Many files are posted to Usenet with obfuscated names. The original file names are needed to successfully unpack downloaded files. NZBGet has built-in automatic renaming. Renaming (par-renaming and rar-renaming) is a post-processing stage where original file names are restored using data from par2-files or meta-information from rar-files. Currently downloading and renaming are separated, which makes the code simpler and more maintainable.

This issue is to investigate and experiment with built-in automatic file renaming during downloading. This feature is a prerequisite for unpack during downloading which is the ultimate goal.

Steps to rename obfuscated files as they are downloaded:

  • download first article of each file included in nzb;
  • analyse first kilobytes of each file to determine which of the file is a par2-file;
  • using first articles calculate 16k hashes for each downloaded file;
  • download one par2-file completely;
  • load par2-file and read 16k hashes;
  • rename already completed files if there are any;
  • set correct names for incomplete files;
  • let download continue with correct names.

Advanced version also needs:

  • support multiple par-sets:
    • detect par-sets;
    • download one par2-file for each par-sets;
    • use every loaded par2-files for renaming;
  • rename par2-files too;
  • properly handle connection interruptions;
  • handle partially downloaded (unnecessary) par2-files (.out.tmp-files):
    • do not leave .out.tmp-files for successful downloads;
    • par-rename and par-check should play nice with partially downloaded files (.out.tmp-files);
    • force cache flushing for partially downloaded files before post-processing;
  • save and restore renaming state if the program was reloaded before renaming completed;
  • new event for queue-scripts; to send after renaming.

@hugbug hugbug added the feature label Apr 16, 2017

@hugbug hugbug added this to the v19 milestone Apr 16, 2017

hugbug added a commit that referenced this issue Apr 22, 2017

hugbug added a commit that referenced this issue Apr 22, 2017

#362: downloading first article of each file
, then other articles, when option “DirectRename” is active.

hugbug added a commit that referenced this issue Apr 22, 2017

#362: fixed restoring of partial state
when direct write was disabled

hugbug added a commit that referenced this issue Apr 22, 2017

hugbug added a commit that referenced this issue Apr 22, 2017

#362: functional tests for direct renamer
(new tests currently fail)

hugbug added a commit that referenced this issue Apr 22, 2017

hugbug added a commit that referenced this issue Apr 22, 2017

hugbug added a commit that referenced this issue Apr 23, 2017

#362: unpause par2-file
as we need it to download first and ExtraPriority-flag doesn’t work on
paused files

hugbug added a commit that referenced this issue Apr 24, 2017

hugbug added a commit that referenced this issue Apr 24, 2017

#362: handling of download interruptions
if they happen during downloading of first articles

hugbug added a commit that referenced this issue Apr 25, 2017

hugbug added a commit that referenced this issue Apr 28, 2017

#362: proper rename on completion
Fixed: if the file was renamed during finalizing stage (completion) the
file may not be properly renamed.

hugbug added a commit that referenced this issue Apr 28, 2017

#362: discard partially downloaded par2-files
to solve issues with par-checker, par-renamer and avoid saving state
for these files.

hugbug added a commit that referenced this issue Apr 28, 2017

#362: redesigned renaming
Spread RenameInfo into FileInfo and NzbInfo to make the state easier to
save on disk

hugbug added a commit that referenced this issue May 1, 2017

#362: new queue event after direct rename
Queue-event NZB_NAMED, sent after the inner files are renamed

@hugbug hugbug closed this May 1, 2017

@hugbug hugbug referenced this issue May 3, 2017

Closed

Unpack during downloading #371

8 of 8 tasks complete

hugbug added a commit that referenced this issue May 16, 2017

hugbug added a commit that referenced this issue May 23, 2017

#362: prevent repeating of error messages
If a file got lost before cache flushing.

hugbug added a commit that referenced this issue May 23, 2017

#362: discard unneeded data after direct rename
Now also discarding data when download completes without direct rename
being able to process files (due to download errors).

hugbug added a commit that referenced this issue Oct 9, 2017

hugbug added a commit that referenced this issue Oct 9, 2017

#362: downloading first article of each file
, then other articles, when option “DirectRename” is active.

hugbug added a commit that referenced this issue Oct 9, 2017

#362: fixed restoring of partial state
when direct write was disabled

hugbug added a commit that referenced this issue Oct 9, 2017

hugbug added a commit that referenced this issue Oct 9, 2017

#362: functional tests for direct renamer
(new tests currently fail)

hugbug added a commit that referenced this issue Oct 9, 2017

hugbug added a commit that referenced this issue Oct 9, 2017

hugbug added a commit that referenced this issue Oct 9, 2017

#362: unpause par2-file
as we need it to download first and ExtraPriority-flag doesn’t work on
paused files

hugbug added a commit that referenced this issue Oct 9, 2017

hugbug added a commit that referenced this issue Oct 9, 2017

#362: handling of download interruptions
if they happen during downloading of first articles

hugbug added a commit that referenced this issue Oct 9, 2017

hugbug added a commit that referenced this issue Oct 9, 2017

#362: proper rename on completion
Fixed: if the file was renamed during finalizing stage (completion) the
file may not be properly renamed.

hugbug added a commit that referenced this issue Oct 9, 2017

#362: discard partially downloaded par2-files
to solve issues with par-checker, par-renamer and avoid saving state
for these files.

hugbug added a commit that referenced this issue Oct 9, 2017

#362: redesigned renaming
Spread RenameInfo into FileInfo and NzbInfo to make the state easier to
save on disk

hugbug added a commit that referenced this issue Oct 9, 2017

#362: new queue event after direct rename
Queue-event NZB_NAMED, sent after the inner files are renamed

hugbug added a commit that referenced this issue Oct 9, 2017

hugbug added a commit that referenced this issue Oct 9, 2017

#362: prevent repeating of error messages
If a file got lost before cache flushing.

hugbug added a commit that referenced this issue Oct 9, 2017

#362: discard unneeded data after direct rename
Now also discarding data when download completes without direct rename
being able to process files (due to download errors).

hugbug added a commit that referenced this issue Oct 9, 2017

#362, #382: fixed crash during direct rename
which may happen if download errors occurred.

hugbug added a commit that referenced this issue Oct 9, 2017

#362, #382: fixed crash during direct rename
which may happen if articles were yEncoded with less than 68 character
length.

hugbug added a commit that referenced this issue Oct 9, 2017

hugbug added a commit that referenced this issue Oct 9, 2017

#362, #382: articles yEncoded with less than 68 characters
are now correctly processed by direct rename.

hugbug added a commit that referenced this issue Oct 9, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment