Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runJobs.php / RefreshJob and the `procs` option #927

Closed
jongfeli opened this Issue Mar 27, 2015 · 19 comments

Comments

Projects
None yet
3 participants
@jongfeli
Copy link
Contributor

commented Mar 27, 2015

When testing SMW performance with the different PHP caching solutions I noticed some "strange" behavior when running runJobs.php. I was experimenting with running runJobs.php with the --conf and --procs options (linux and unix only) on Ubuntu to see what is does to increase performance. I did not understand what happened then so I tried to simulate what happened at home.

I setup a SMW test wiki with smw.content.annotation.import.xml and with the help of SemanticForms, 100 pages where automatically created.

After that press Start updating data on Special:SMWAdmin and then run:

time php runJobs.php --conf=/var/www/html/mediawiki-1.24.1/LocalSettingsrunJobs.php --procs=4

It started running with 4 processes (threads) but shortly after that 3 of them "died" and the jobs where finished on one process (thread). I did this over and over again and the same thing happened again and again. It seems that after or before SMW\RefreshJob Special:SMWAdmin spos=1 prog=0 rc=2 run=2 STARTING only one thread continues to finish processing the jobs.

Example 1:

2015-03-26 16:07:58 SMW\RefreshJob Special:SMWAdmin spos=541 prog=1.0226843100189 rc=2 run=1 STARTING
2015-03-26 16:07:59 SMW\RefreshJob Special:SMWAdmin spos=541 prog=1.0226843100189 rc=2 run=1 t=8 good
2015-03-26 16:07:59 SMW\RefreshJob Special:SMWAdmin spos=1 prog=0 rc=2 run=2 STARTING
2015-03-26 16:07:59 SMW\RefreshJob Special:SMWAdmin spos=1 prog=0 rc=2 run=2 t=46 good
2015-03-26 16:07:59 SMW\UpdateJob Property:Has_type STARTING
2015-03-26 16:07:59 SMW\UpdateJob Property:Has_type t=3 good
2015-03-26 16:07:59 SMW\UpdateJob Property:Has_wattage STARTING
2015-03-26 16:07:59 SMW\UpdateJob Property:Has_wattage t=33 good

Example 2:

2015-03-26 16:06:08 SMW\RefreshJob Special:SMWAdmin spos=521 prog=0.98487712665406 rc=2 run=1 t=8 good
2015-03-26 16:06:08 SMW\RefreshJob Special:SMWAdmin spos=1 prog=0 rc=2 run=2 STARTING
2015-03-26 16:06:08 SMW\RefreshJob Special:SMWAdmin spos=541 prog=1.0226843100189 rc=2 run=1 t=6 good
2015-03-26 16:06:08 SMW\RefreshJob Special:SMWAdmin spos=1 prog=0 rc=2 run=2 t=36 good
2015-03-26 16:06:08 SMW\RefreshJob Special:SMWAdmin spos=21 prog=0.039697542533081 rc=2 run=2 STARTING

If you halt the process with ctrl-c and restart it it will restart with 4 threads and continue to do so until it is finished. I am not sure if this is a problem in runJobs.php but I noticed that jobs get added to the job table during the run. Could it be that SMW is actually restarting runJobs.php with only one process at a certain point? Being able to run runJobs.php in multiple processes increases performance tremendously. Triggered by [Introduce SemanticDataCache #820], could the same be done with rebuildData.php? It does not support the --procs parameter but is it possible to make it so that it does, to increase performance?

@mwjames mwjames changed the title runJobs.php performance problem runJobs.php / RefreshJob and the `procs` option Mar 27, 2015

@mwjames mwjames added the question label Mar 27, 2015

@mwjames

This comment has been minimized.

Copy link
Contributor

commented Mar 27, 2015

with the --conf and --procs options (linux and unix only)

An old bug [0] had the same interest in using this option

It started running with 4 processes (threads) but shortly after that 3 of them "died" and the jobs where finished on one process (thread). I did this over and over again and the same thing happened again and again. It seems that after or before SMW\RefreshJob Special:SMWAdmin spos=1 prog=0 rc=2 run=2 STARTING only one thread continues to finish processing the jobs.

I never tried to run it with a specific number of --procs and since I'm not running any lux (as noted by the ForkController the "class requires the posix and pcntl extensions"), I'm unable to verify or yet alone to analyze the issue.

If you halt the process with ctrl-c and restart it it will restart with 4 threads and continue to do so until it is finished. I am not sure if this is a problem in runJobs.php but I noticed that jobs get added to the job table during the run.

Interesting.

Could it be that SMW is actually restarting runJobs.php with only one process at a certain point?

This could well be.

It seems that after or before SMW\RefreshJob Special:SMWAdmin spos=1 prog=0 rc=2 run=2 STARTING only one thread continues to finish processing the jobs.

Details on how jobs are invoked/created recursively can be found at [1].

Triggered by [Introduce SemanticDataCache #820], could the same be done with rebuildData.php? It does not support the --procs parameter but is it possible to make it so that it does, to increase performance?

rebuildData.php is not running a Job ( --procs belongs to the jobsclass, independent from whether it is scheduled as job) but re-uses $updatejob = new UpdateJob( $title ); $updatejob->run(); or $this->store->refreshData( $id, 1, false, false ); [2] directly without any job scheduling involved.

[0] https://phabricator.wikimedia.org/T47321

[1] https://github.com/SemanticMediaWiki/SemanticMediaWiki/blob/master/src/MediaWiki/Jobs/RefreshJob.php

[2] https://github.com/SemanticMediaWiki/SemanticMediaWiki/blob/master/src/Maintenance/DataRebuilder.php#L229

See also

@kghbln Ever tried to use --procs?

@kghbln

This comment has been minimized.

Copy link
Member

commented Apr 1, 2015

@kghbln Ever tried to use --procs?

Not for this script since I was aware of the issue on bugzilla. Being able to do multiple procs willl indeed be a nice feature

@jongfeli

This comment has been minimized.

Copy link
Contributor Author

commented Apr 3, 2015

Independent of SMW, when you use --procs=.. with runJobs.php to forcefully update your wiki after a template change then, depending on the speed of your storage back-end, the performance increase can be significant. I did some tests which I am not able to post at the moment because I don't have "them" with me, I will do that later today.

It is possible to forcefully update your wiki and use the --procs=.. option with runJobs.php after you pressed Start updating data on Special:SMWAdmin but at a certain point it gets reset to just a single process.

@mwjames you noted that it could very well be that SMW is restarting runJobs.php with only a single process . Is this a "fact" or an assumption? If this is the case could SMW restart runJobs.php with the original amount of procs set when runJobs.php started?

@mwjames what do you mean with as noted by the ForkController where I can see that you are not using any lux? Because you are not using lux but the rest of the world is :) I am more then happy to provide you with any information you need, if you want.

@mwjames

This comment has been minimized.

Copy link
Contributor

commented Apr 3, 2015

you noted that it could very well be that SMW is restarting runJobs.php with only a single process . Is this a "fact" or an assumption?

As I said, I don't run Linux/Unix therefore I can't test the procs option (it works only on Linux/Unix) ergo I can't make an analysis.

If I have to make a guess then I would point to [0] where I believe that the parameter array needs an additional "procs" => $this->hasParameter( 'procs' ) ? $this->getParameter( 'procs' ) : 1 entry (but please keep this in mind it isn't tested, it is a conjecture based on a review of the code).

what do you mean with as noted by the ForkController

This is the part that handles/forks processes in MW during the runJobs execution.

where I can see that you are not using any lux? Because you are not using lux but the rest of the world is :)

vide supra

I am more then happy to provide you with any information you need, if you want.

You can try the suggestion from above, and report back on whether it works or not (and send an PR if it works) .

[0] https://github.com/SemanticMediaWiki/SemanticMediaWiki/blob/master/src/MediaWiki/Jobs/RefreshJob.php#L89

@mwjames

This comment has been minimized.

Copy link
Contributor

commented Apr 13, 2015

@jongfeli Did you try out the suggestion made in #927 (comment)?

@jongfeli

This comment has been minimized.

Copy link
Contributor Author

commented Apr 17, 2015

Hello @mwjames. My apologies for the delayed reaction but I have been quite busy lately (we all are). A couple of weeks ago I did try your suggested code but then it did not work. It could be that it simply did not work or that I don't know what I am doing, I think it is the last :). No matter what value we use in the procs parameter it always shows the default of 1. But I did not take enough time to understand how it works and what happens. I need to do that first.

In short what I did:

    /**
     * @param $spos start index
     */
    protected function refreshData( $spos ) {
        Profiler::In();
        $run  = $this->hasParameter( 'run' ) ? $this->getParameter( 'run' ) : 1;
        $prog = ApplicationFactory::getInstance()->getStore()->refreshData( $spos, 20, $this->getNamespace( $run ) );
        $procs = $this->hasParameter( 'procs' ) ? $this->getParameter( 'procs' ) : 1;

        if ( $spos > 0 ) {
            $this->createNextJob( array(
                'spos' => $spos,
                'prog' => $prog,
                'rc'   => $this->getParameter( 'rc' ),
                'run'  => $run,
                'procs'  => $procs 
            ) );
        } elseif ( $this->getParameter( 'rc' ) > $run ) { // do another run from the beginning
            $this->createNextJob( array(
                'spos' => 1,
                'prog' => 0,
                'rc'   => $this->getParameter( 'rc' ),
                'run'  => $run + 1,
                'procs'  => $procs 
            ) );
        }
        Profiler::Out();
        return true;
    }
@mwjames

This comment has been minimized.

Copy link
Contributor

commented Apr 20, 2015

A couple of weeks ago I did try your suggested code but then it did not work. It could be that it simply did not work or that I don't know what I am doing, I think it is the last :). No matter what value we use in the procs parameter it always shows the default of 1.

Code looks good.

As I said, it was a guess and since on windows I only get gibberish, I can't really make any other suggestion. Anyone with a lux box and some time on hand should try to debug this as to where the initial (e.g. --procs=4) becomes --procs=1.

$ php ../../maintenance/runJobs.php --procs=4

Notice: Use of undefined constant SIGFPE - assumed 'SIGFPE' in ...\maintenance\runJobs.php o
n line 62

Call Stack:
    0.0013     140448   1. {main}() ...\maintenance\runJobs.php:0
    0.0108     358736   2. require_once('...\maintenance\doMaintenance.php') ...\maintenance\runJobs.php:95
    0.7639    9401480   3. RunJobs->execute() ...\maintenance\doMaintenance.php:101
@jongfeli

This comment has been minimized.

Copy link
Contributor Author

commented Jul 25, 2015

I tried picking this up again but stumbled on something I already noticed before. On our wiki pages get created automatically with rebuildDate.php (previously called SMW refreshData.php) and runJobs.php. We use the ExternalData extension to load customers into the wiki. When a customer is new the page will then be created automatically. This worked fine but stopped working some time ago, not sure when.

Testing this in my home wiki the same thing happens. Example with a property page called Property:Testname with the following content:

* Has type:[[Has type::Page]]
* [[Creates pages with form::Testcaseautofill]]

== Test pages ==
* [[Testname::Testcase1]]
* [[Testname::Testcase2]]
* [[Testname::Testcase3]]
* [[Testname::Testcase4]]

After saving the page and running php runJobs.php the 4 pages get created automatically.

When I then delete Testcase1 and rebuild with:

php rebuildData.php --no-cache -v

and

php runJobs.php

The page Testcase1 does not get created. The property Testname actually has the value Testcase1 but with a red link. Even when I use the Data repair and upgrade function and run php runJobs.php the page Testcase1 does not get created. The only way to get the page to be created automatically is to refresh the Property:Testname page and run php runJobs.php.

I am pretty sure that php rebuildData.php or SMW refreshData.php added pages to be created to the MW job Que which you could then "empty" by using runJobs.php. I am not sure why this is not working anymore. It seems that php rebuildData.php is "ignoring" [[Creates pages with form::Testcaseautofill]] or SF as a whole or SF is not doing its "job".

Test done on MW 1.24.1, SMW dev-master and SF 3.3.1

@jongfeli

This comment has been minimized.

Copy link
Contributor Author

commented Jul 31, 2015

Ok I did some research and the above "functionality" stopped working in SMW 1.9.2. In this release the move was made from SMW_refreshData.php to rebuildData.php. I tested this on MW 1.23.9 and different SF versions, 2.8 and master.

This basically means that since SMW 1.9.2 it is not possible to create pages via the command line anymore with the [[Creates pages with form::...]] functionality of SF, rebuildData.php and runJobs.php (#244). The only way to be able to automatically create these pages in the example above is to refresh the Property:Testname page in the browser.

I do not know if this was intended (foreseen) or that this is a bug. In my opinion the use of SF's [[Creates pages with form::...]] to create pages automatically without any user intervention is very powerful and useful functionality. Should I create a separate issue for this?

@mwjames

This comment has been minimized.

Copy link
Contributor

commented Jul 31, 2015

I do not know if this was intended (foreseen) or that this is a bug. In my opinion the use of SF's [[Creates pages with form::...]] to create pages automatically without any user intervention is very
To be honest, I have no idea what SF does or how it is connected to rebuildData.php.

rebuildData.php / #244 does not directly work with (or discriminate against) any specific property or property type [0] therefore I can't see the connection to #244.

[0] https://github.com/wikimedia/mediawiki-extensions-SemanticForms/blob/a4c325486bff2a4406fb687ac0e4967ce6c0339e/includes/SF_Utils.php#L185

It seems that php rebuildData.php is "ignoring" [[Creates pages with form::Testcaseautofill]]

rebuildData.php is executing a Parser::parse (broadly speaking) and everything that is annotated and returned (by executing the InTextAnnotationParser) to the ParserOutput -> SemanticData object is being fed to the UpdateStore process.

it is not possible to create pages via the command line anymore with the [[Creates pages with form::...]]

@kghbln Not sure what this is about but rebuildData.php's job is primarily to update the Store with annotations found during the page re-parse process and no to create pages.

powerful and useful functionality. Should I create a separate issue for this?

If rebuildData.php works for what it is specified namely re-parse a page, collect its annotations, and update the Store (as its main functionality) isn't diminished then I can't see the connection to SMW.

[[Creates pages with form::...]] functionality of SF, rebuildData.php ... to be able to automatically create these pages

Something as above is not part of the specification of what rebuildData.php is expected to do therefore I can't classify this as a bug but if by any chance you have a diff/PR (without introducing some SF dependency) to recover this feature that would certainly be 👍 .

EDIT: (lashing out here a bit) If by any offbeat chance this feature was intended by SF but never codified in SMW to be supported then it can not be expected to work as undocumented feature through releases.

If a hook was provided to make such feature available for the implementation side of SF that would turn the argument but such hook was not defined in/before 1.9.2 nor has it been in 2.3 therefore the functionality of rebuildData.php is as described and any implicit behaviour should not be taken as explicit statement of supported functionality (intended behaviour is required to be codified and to be tested).

@mwjames

This comment has been minimized.

Copy link
Contributor

commented Sep 3, 2015

Using [0] as guideline, the video [1] could not verify the issue as described in [2]. Given the evidence [1], I'm inclined to close this issue.

[0] https://www.mediawiki.org/wiki/Extension_talk:Semantic_Forms#Automatic_page_creation_.22broken.22_with_SMW.27s_php_rebuildData.php

[1] https://youtu.be/jsHhaeyrduw (just made the video as evidence)

[2] #927 (comment)

@jongfeli

This comment has been minimized.

Copy link
Contributor Author

commented Sep 4, 2015

A appreciate the time you took to test this, thanks for that 👍 . I can replicate what you did and by doing that and looking at the video you can actually see it does not work. After you run php runJobs.php in the video you refresh the Recent changes page by pressing Go. When it refreshes you see that the page Testcase1 was not being recreated.

You then press on Property:Testname in the Recent changes list and by doing that SF gets triggered as soon as the page Property:Testname is loaded and puts the page to be created (Testcase1) in the MW job queue. This is not shown on Property:Testname because the page creation is still in the job queue.

You then refresh Property:Testname and depending on the value for $wgJobRunRate (default = 1) the refresh triggers the page creation for Testcase1. This means that rebuildDate.php does not trigger SF where SMW refreshData.php in the past did.

Like you rightly said before I don't know if it should but because of this in some scenarios it is not possible any more to automatically generate pages on a wiki without user intervention. Real world example is data loaded via the External data extension that triggers page creation. In the example below customer page creation is automated, well it was automated :) for the Property:Customer

{{#get_db_data:db=Externalmachine|from=customer|order by=Name|data=Name=Name}}
{{#for_external_table:<nowiki/>
* [[Customer::{{{Name}}}]]
}}

I do not know if this is a very well know use case but in my opinion being able to create pages on a Semantic wiki with data form any source you want (XML, Excel, SAP, SQL Server ... etc) and not actually touching the wiki itself is a very powerful tool. It would be nice if this "functionality" started working again but I do not know where it should be solved, in SF or SMW. That is the reason I posted this on the Sematic Forms talk list because I am not sure if Yaron is reading "everything" on GitHub and I wanted him to know about this.

As for closing this issue, the actual issue is not about this but about the --proc option under Linux. That is my fault 👎 by mixing them up. The --proc issue is not solved yet, I need to spend some time on that to figure out what is happening.

@mwjames

This comment has been minimized.

Copy link
Contributor

commented Sep 4, 2015

A appreciate the time you took to test this, thanks for that 👍 . I can replicate what you did and by doing that and looking at the video you can actually see it does not work. After you run php runJobs.php in the video you refresh the Recent changes page by pressing Go. When it refreshes you see that the page Testcase1 was not being recreated.

It doesn't matter whether you press refresh or not.The page was deleted as seen in the video and after rebuildeData \ runJobs (the redelink is a matter of the PasrserCache and not a matter of a valid revision) was executed the page was re-created as seen in the video (at the end of the video otherwise when revisiting Testcase1 you would see a delete notice when viewing a page).

The claim that page Testcase1 doesn't get recreated after a delete action after running rebuildeData \ runJobs is not correct. (the second video [0] shows it more clearly)

[0] http://youtu.be/2XVN4c2JRNs

@jongfeli

This comment has been minimized.

Copy link
Contributor Author

commented Sep 4, 2015

OK, I could not see what you did on the top of your screen but you probably refreshed the page Testcase1 in between so I did the same thing with the same result, the page gets recreated.

The problem is that this only happens because we refreshed the page Testcase1. This action also puts the Testcase1 page in the MW job queue to be recreated. But if we would run rebuildData.php and runJobs.php and we would not refresh the page Testcase1 or Property:Testname the page will not get recreated by runJobs.php.

In the past, after SMW refreshData.php did its magic the runJobs.php run would (re)create pages on its own, this means during the initial runJobs run something like below would show:

2015-09-04 10:38:15 createPage Testcase1 user_id=1 page_text= (id=793) STARTING
2015-09-04 10:38:15 createPage Testcase1 user_id=1 page_text= (id=793) t=176 good

There was no user intervention needed via the website (browser) all magic happened via command line. This means that if you would only look at Recent changes after SMW refreshData.php and runJobs.php had run the page was already (re)created.

If you test this like this please make sure you empty the job queue before you delete Testcase1.

@mwjames

This comment has been minimized.

Copy link
Contributor

commented Sep 4, 2015

If you test this like this please make sure you empty the job queue before you delete Testcase1.

I'm actually finished testing with the last video. SMW does what is expected for and whether SF creates something like "createPage Testcase1" or not isn't really a concern of SMW core.

@jongfeli

This comment has been minimized.

Copy link
Contributor Author

commented Sep 4, 2015

I understand and you are probably right. This would mean that SMW refreshData.php probably parsed the page more like a user would but rebuildData.php is not doing that any more. But @mwjames you probably know that better then anybody. In my opinion rebuildData.php is such an important tool to build, rebuild or refresh a SMW Wiki that it should also take all extensions into account that are used on the wiki.

We can discus endlessly where the "problem" lies but I am not a PHP developer so I can not fix it or add it to be "real" functionality between SMW and SF. As it is the automated page creation is supported by SF not by SMW, and it worked great but in my opinion it could well be functionality that is available in SMW core.

@mwjames

This comment has been minimized.

Copy link
Contributor

commented Sep 4, 2015

This would mean that SMW refreshData.php probably parsed the page more like a user would but rebuildData.php is not doing that any more.

There is (was) no change in how content is parsed. The script selects an ID which is then forwarded to the UpdateJob with mode run (which means to execute the Job immediately). Neither rebuildData nor refreshData is (was [0]) responsible for parsing any data (that's the job of UpdateJob).

image

SMW Wiki that it should also take all extensions into account that are used on the wiki.

This happens when extensions provide unit/integration tests that can be executed together to see whether a behavioural change did occur. Depending on the extensions maintainer such tests are provided or not .

[0] https://github.com/SemanticMediaWiki/SemanticMediaWiki/blob/1.9.1/maintenance/SMW_refreshData.php#L173-L175

@jongfeli

This comment has been minimized.

Copy link
Contributor Author

commented Sep 4, 2015

OK, @mwjames thanks for your time and patience.

@mwjames

This comment has been minimized.

Copy link
Contributor

commented Mar 12, 2016

Closing this as no actionable tasks could be assigned to drive this issue forward.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.