Fixing the reports summaries #9

Closed
preaction opened this Issue Apr 30, 2016 · 8 comments

Comments

Projects
None yet
1 participant
@preaction
Member

preaction commented Apr 30, 2016

The reports summaries were offline for quite a while. This affected the version summary bars on the reports website, and the release SQLite database (which is generated from the same data, but using a different process).

We've fixed it, but we should track it here to ensure everything's wrapped up.

Related issues:

@preaction

This comment has been minimized.

Show comment
Hide comment
@preaction

preaction Sep 3, 2016

Member

Everything here seems cleared up, so I'm closing this. Writing a better API for the reports summaries is a different issue.

Member

preaction commented Sep 3, 2016

Everything here seems cleared up, so I'm closing this. Writing a better API for the reports summaries is a different issue.

@preaction preaction closed this Sep 3, 2016

@preaction

This comment has been minimized.

Show comment
Hide comment
@preaction

preaction Sep 4, 2016

Member

And because I said it, it broke again. Reopening.

Member

preaction commented Sep 4, 2016

And because I said it, it broke again. Reopening.

@preaction preaction reopened this Sep 4, 2016

@preaction

This comment has been minimized.

Show comment
Hide comment
@preaction

preaction Sep 11, 2016

Member

This is generated by /var/www/reports/toolkit/reports-release.sh (which is called from cron) which calls /home/barbie/bin/process-controller.pl (not on CPAN that I can see) to manage a process using /var/www/reports/toolkit/reports-release.ini. This runs perl reports-release.pl --update from the /var/www/reports/toolkit directory, after killing all existing instances of reports-release.pl using killall. This is logged to /var/www/reports/toolkit/logs/release-run.log.

The log shows a bunch of errors from the database, but in HTML. So it must be using Labyrinth to do its work. The log also shows that it stopped processing new records around April 25 (2016/04/25 16:31:41 .. summary max=67995474, data max=67995474 which was the last increase, we are now at 73000000 records). This was during the QA Hackathon, so it doesn't seem to have worked since then...

These records are in the cpanstats.release_summary table. I need to find what updates this table.

Member

preaction commented Sep 11, 2016

This is generated by /var/www/reports/toolkit/reports-release.sh (which is called from cron) which calls /home/barbie/bin/process-controller.pl (not on CPAN that I can see) to manage a process using /var/www/reports/toolkit/reports-release.ini. This runs perl reports-release.pl --update from the /var/www/reports/toolkit directory, after killing all existing instances of reports-release.pl using killall. This is logged to /var/www/reports/toolkit/logs/release-run.log.

The log shows a bunch of errors from the database, but in HTML. So it must be using Labyrinth to do its work. The log also shows that it stopped processing new records around April 25 (2016/04/25 16:31:41 .. summary max=67995474, data max=67995474 which was the last increase, we are now at 73000000 records). This was during the QA Hackathon, so it doesn't seem to have worked since then...

These records are in the cpanstats.release_summary table. I need to find what updates this table.

@preaction

This comment has been minimized.

Show comment
Hide comment
@preaction

preaction Sep 11, 2016

Member

perl reports-release.pl --update runs the Update sub from Labyrinth::Plugins::CPAN::Release. This pulls the summary max from SELECT MAX(id) FROM cpanstats.release_summary and the data max from SELECT MAX(id) FROM cpanstats.release_data. So, it seems the release_summary gets updated from release_data. So now what updates release_data?

Member

preaction commented Sep 11, 2016

perl reports-release.pl --update runs the Update sub from Labyrinth::Plugins::CPAN::Release. This pulls the summary max from SELECT MAX(id) FROM cpanstats.release_summary and the data max from SELECT MAX(id) FROM cpanstats.release_data. So, it seems the release_summary gets updated from release_data. So now what updates release_data?

@preaction

This comment has been minimized.

Show comment
Hide comment
@preaction

preaction Sep 11, 2016

Member

The release_data table is created by /var/www/reports/toolkit/reports-release-create.sh which manages a process that eventually calls perl reports-release.pl --create. This needs to be done on a cron job in order to keep the release data in sync.

Member

preaction commented Sep 11, 2016

The release_data table is created by /var/www/reports/toolkit/reports-release-create.sh which manages a process that eventually calls perl reports-release.pl --create. This needs to be done on a cron job in order to keep the release data in sync.

@preaction

This comment has been minimized.

Show comment
Hide comment
@preaction

preaction Sep 11, 2016

Member

A new line is added to the barbie user's crontab which runs the /var/www/reports/toolkit/reports-release-create.sh about 90 minutes before /var/www/reports/toolkit/reports-release.sh, which should be enough time for the release to be created.

It is currently running, but it has 5 million records to get through this first time.

We'll give this a week and check on it again.

Member

preaction commented Sep 11, 2016

A new line is added to the barbie user's crontab which runs the /var/www/reports/toolkit/reports-release-create.sh about 90 minutes before /var/www/reports/toolkit/reports-release.sh, which should be enough time for the release to be created.

It is currently running, but it has 5 million records to get through this first time.

We'll give this a week and check on it again.

@preaction

This comment has been minimized.

Show comment
Hide comment
@preaction

preaction Sep 13, 2016

Member

The script that moves data from cpanstats.release_summary to a SQLite database is /opt/projects/cpantesters/release/bin/release.pl which is executed by /opt/projects/cpantesters/autorun-back3.sh. The release.pl creates the SQLite database in /opt/projects/cpantesters/release/data/release.db and then the autorun-back3.sh copies the database to /opt/projects/cpantesters/db/release.db, then copies it twice more to /opt/projects/cpantesters/dbx/release.db to compress one with gzip and the other with bzip2. These compressed versions are then moved to /var/www/cpandevel/release which is available at http://devel.cpantesters.org/release/release.db.gz and http://devel.cpantesters.org/release/release.db.bz2.

I found an issue in the reports-release.ini and reports-release-create.ini that was preventing one of them from running while the other was running. They were both looking for just the string reports-release.pl in the ps aux output, which they both use (but with different flags). I added --update and --create to their search so they shouldn't find each other. This might be a bad thing to fix, considering the load on the server, but I fixed it anyway so I can see the data flowing to the release_summary table.

Member

preaction commented Sep 13, 2016

The script that moves data from cpanstats.release_summary to a SQLite database is /opt/projects/cpantesters/release/bin/release.pl which is executed by /opt/projects/cpantesters/autorun-back3.sh. The release.pl creates the SQLite database in /opt/projects/cpantesters/release/data/release.db and then the autorun-back3.sh copies the database to /opt/projects/cpantesters/db/release.db, then copies it twice more to /opt/projects/cpantesters/dbx/release.db to compress one with gzip and the other with bzip2. These compressed versions are then moved to /var/www/cpandevel/release which is available at http://devel.cpantesters.org/release/release.db.gz and http://devel.cpantesters.org/release/release.db.bz2.

I found an issue in the reports-release.ini and reports-release-create.ini that was preventing one of them from running while the other was running. They were both looking for just the string reports-release.pl in the ps aux output, which they both use (but with different flags). I added --update and --create to their search so they shouldn't find each other. This might be a bad thing to fix, considering the load on the server, but I fixed it anyway so I can see the data flowing to the release_summary table.

@preaction

This comment has been minimized.

Show comment
Hide comment
@preaction

preaction Sep 17, 2016

Member

We've now reached 72,000,000 reports processed for the release data. I've checked the release.db and it contains updated reports. However, metacpan does not seem to have loaded the release: https://metacpan.org/release/DROLSKY/Params-CheckCompiler-0.07 definitely has reports in the release database, but the reports are not presently showing. Going to update them on the status and see if it's on their end.

From our end this looks fixed. Closing.

Member

preaction commented Sep 17, 2016

We've now reached 72,000,000 reports processed for the release data. I've checked the release.db and it contains updated reports. However, metacpan does not seem to have loaded the release: https://metacpan.org/release/DROLSKY/Params-CheckCompiler-0.07 definitely has reports in the release database, but the reports are not presently showing. Going to update them on the status and see if it's on their end.

From our end this looks fixed. Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment