Skip to content

Commit

Permalink
commit every 200 scan results to the database an error would make the…
Browse files Browse the repository at this point in the history
… scan of big libraries impossible
  • Loading branch information
Rello committed Aug 24, 2019
1 parent df8ef00 commit 7d2a204
Showing 1 changed file with 4 additions and 0 deletions.
4 changes: 4 additions & 0 deletions lib/Controller/ScannerController.php
Original file line number Diff line number Diff line change
Expand Up @@ -173,6 +173,10 @@ public function scanForAudios($userId = null, $output = null, $scanstop = null)
if ($this->timeForUpdate()) {
$this->updateProgress($counter, $audio->getPath(), $output);
}
if ($counter % 200 == 0) {
$this->DBController->commit();
$output->writeln("Status committed to database", OutputInterface::VERBOSITY_VERBOSE);
}
}

$output->writeln("Start processing of <info>stream files</info>", OutputInterface::VERBOSITY_VERBOSE);
Expand Down

4 comments on commit 7d2a204

@Rello
Copy link
Owner Author

@Rello Rello commented on 7d2a204 Aug 24, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mmatous
Copy link
Contributor

@mmatous mmatous commented on 7d2a204 Aug 24, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you commit, you need to begin transaction again. This just kills performance and throws exception when next commit is due.

Also 200 seems like being way too careful. psy-q had misconfigured server that used 128MB instead of 512MB RAM (NC minimum) and got into high thousands. I'll do some tests with 128 and let you know safe threshold.

@Rello
Copy link
Owner Author

@Rello Rello commented on 7d2a204 Aug 24, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. Will fix my mistake. Worked for me but might be related to sqlite on the test-Nc.

I think its not only a memory threshold issue. I had several issues where users had faulty files which could not be caught by getID3. Or php timeouts; or issues with SMB.

The workaround was „do it several times“

Why do you think eg 200 is bad?
For a 500track libraries it reduces DB operations to 3. compared you your suggested 1. compared to my old coding of ~600.

Or do I miss something?
Your suggestion with the commit was perfect...

@Rello
Copy link
Owner Author

@Rello Rello commented on 7d2a204 Aug 25, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 more ideas:

  • we offer an -occ parameter to set this level during an occ scan
  • we use a relative value like 10% of audios-count

completely without I don´t feel comfortable somehow

Please sign in to comment.