Progressive rendering not possible

[ctrlaltca]

From google...@pcforum.hu on March 13, 2010 21:56:53

Seems like progressive rendering (ie. rendering smaller parts of the page
and sending them to client right after completed, one after the other,
instead of sending the whole page at once, after rendering of the whole
page has completed) is currently not possible with Prado. This is because
even the outermost container of all Prado pages, the TForm component
itself is buffering all of its contents and then emitting it at once.

Looking at TForm::render() I see no obvious reason for that, except being
a hack to work around poorly written/incosistent code in the framework and
in the application, that isn't/aren't able to property determine their
script/field usage prior to the actual rendering, and therefore do that
(ie. register (begin)scripts and fields) in the render() method. In my
opinion this is just the result of bad design/coding practices, as all
that information should be either determined and be present already in the
PreRender() state and/or the field declarations and script references
should be emitted inline in the render() process (as <script> and
block are allowed to appear anywhere in the page anyway) - but all that
could be perfectly well done without halting/stalling the output of
already rendered page parts due to the need to buffer all output until the
whole page has finished, just so we can be 100% sure that even
malformed/badly written code (which determines and/or communicated it's
need for scripts and hidden fields communicates only in the render phase
to the framework) will render properly. That's not the way to go IMHO. The
standard Prado framework code should be cleaned up (ie. all beginscript
and hidden field registeration - if there are any - should be moved to the
PreRender() methods from render()s), and any attempt to register new
beginscripts or post fields by application code in the render phase of the
page should be "honored" with an exception in the appropriate registration
methods.

Progressive rendering is very important in reducing initial response times
and thus in improving user experience (because users will perceive the
page as loading faster, if they have at least something appearing quick,
instead of getting the whole page, but only seconds later).

Prado/3.1.7

Original issue: http://code.google.com/p/prado3/issues/detail?id=235

[ctrlaltca]

From google...@pcforum.hu on March 13, 2010 18:01:49

OK, I just created a rudimentary patch to solve this problem. This should also solve
the problem described in Issue#214 (ie. getting PHP header errors when sending pages
or files which are larger than the PHP output buffer). In order to use the patch and
get progressive rendering capability you have to copy the attached .zip file over an
existing Prado/3.1.7 installation. After that Prado won't practically buffer all
your output (even if you have OutputBuffering disabled) until page rendering is
finished, and emit the entire page only then in a single chunk, but will send the
contents of the page immediately as they become available during
rendering - that means if you did not enabled OutputBuffering.

Also, even if you have OutputBuffering enabled you can still force the sending of
the already rendered parts of the page to the client (which wasn't possible
previously) at specific points by including a new control "<com:TFlushOutput />" in
your template code. You can have any number of those, and whenever rendering reaches
them, they will force-flush any pending output buffered to that point. (For ex. you
might want to flush the output once you rendered the header and logo of your page,
which thus will be immediately visible to your user, and then you can take your time
to render the main column of your page, which can also take multiple seconds for ex.
if you have a large dataset, or even flush over and over after rendering every N
row, and so providing a progressive display to your users.)

Now, in order for all this to work a single rule must be met: prepare all your HTTP
response code (200 for OK, 404 for not found, etc) and Content-Type information (if
you're serving up something other than a text/html web page) up to and inclusive the
PreRender phase of the page cycle, and don't try to alter any of those any more in
the render phase (as parts of your response and the headers might have been already
sent to the client). Also, try to publish all your scripts and hidden fields in the
PreRender phase - if you fail to do so, and try to publish scripts or hidden fields
in the render phase, the patch will put those scripts at the end of the page, before
the endscript (which shouldn't be a problem as long as you use the classes and
functions in those scriptfiles only from endscripts, but might cause problems if you
emit inline javascript in your page's code trying to call into them).

I didn't have time yet to test all standard controls whether they still work
flawlessly with the patch, but they all should work, and even if for whatever reason
they or any 3rd party components don't, they can be most likely easily fixed by just
slightly rearranging their code (moving the asset-publishing code to the PreRender
phase from the render() methods, where it would belong anyway).

I've also included a small .txt file in the .zip that describes the major
implementational changes I made to the libraries.

All feedback appreciated.

Attachment: prado-progressive-rendering-update.zip

[ctrlaltca]

From google...@pcforum.hu on March 13, 2010 18:05:19

OK, forgot to include the .txt. Here it comes.

Attachment: changes.txt

[ctrlaltca]

From google...@pcforum.hu on March 14, 2010 19:02:27

Just fixed an issue (output handlers weren't closed properly, even though the code
did what it was supposed to do). Updated sources attached.

Attachment: prado-progressive-rendering-update.zip

[ctrlaltca]

From google...@pcforum.hu on March 18, 2010 12:28:23

Update to the update: patched versions of the THtmlArea, TDraggable and
TDropContainer controls made compatible with the progressive rendering patch. The
only changes to the controls/files are that script registrations are moved from the
render phase to the prerender phase.

Attachment: prado-progressive-rendering-controls1.zip

[ctrlaltca]

From rojaro@gmail.com on May 30, 2011 03:53:39

Labels: -Milestone-3.2a Milestone-3.1.9

[ctrlaltca]

From rojaro@gmail.com on June 01, 2011 01:04:59

Labels: -Milestone-3.1.9 Milestone-3.1.10

[ctrlaltca]

From rojaro@gmail.com on June 01, 2011 01:09:47

Labels: -Milestone-3.1.10 Milestone-3.2a

[ctrlaltca]

From ctrlal...@gmail.com on June 24, 2011 15:13:27

patch merged to trunk in r2998 , thank you!

Status: Fixed

[ctrlaltca]

From google...@pcforum.hu on June 24, 2011 17:01:36

I've weeded out some more minor bugs in the implementation which caused trouble. See the attached archive. All the files are still based on 3.1.7, not on the latest 3.1.8.

Attachment: progressive-patch-v2.zip

[ctrlaltca]

From google...@pcforum.hu on June 24, 2011 18:03:55

Oh, and btw you did not add the TFlushOutput control in the r2998 patch, which is crucial in enabling page flushes (ie. sending the contents rendered up to that point) at designated places during generation of the page, instead of having flushes occour automatically (and thust mostly unpredictably) at buffer boundaries.

[ctrlaltca]

From google...@pcforum.hu on June 24, 2011 18:43:47

There are also some lines missing from your r2998 patch. You need to add

    $cs=$this->getPage()->getClientScript();

at both line 148 and then at line 161->162 to framework/Web/UI/ActiveControls/TDraggable.php

[ctrlaltca]

From ctrlal...@gmail.com on June 25, 2011 02:09:47

Additional patched have been applied in r3000 ; i had to rework a bit the TDatePicker/TActiveDatePicker one, since it was not working under some configurations.

[ctrlaltca]

From google...@pcforum.hu on November 27, 2011 09:02:43

There's a superflous check/constraint in this patch, regarding the registration of hidden fields. The TClientScriptManager::registerHiddenField() method checks whether form rendering has already begun as doesn't allow registering new hidden fields if it did.

However, since the patch checks also at the end of the form rendering cycle whether there are any not-yet-rendered hidden fields, and emit them if needed, this check in the registration method is not needed, as hidden fields can be very well safely registered while the rendering is in progress. So, the line

    $this->checkIfNotInRender();

should be simply removed from TClientScriptManager::registerHiddenField() in TClientScriptManager.php at around line 563.

[ctrlaltca]

From ctrlal...@gmail.com on November 28, 2011 02:08:13

committed as r3075 , thank you

[ctrlaltca]

From google...@pcforum.hu on May 16, 2012 17:11:05

This patch needs a fix, because the original version does not handle Prado package sets correctly, and fails to include in the page any extra Prado packages that are registered during the render phase, if there were any packages registered already in the prerender phase.

Please find the attached testcase and fix.

Attachment: prado-package-reg-fail-testcase.zip prado-package-reg-fail-fix.zip

[ctrlaltca]

From google...@pcforum.hu on May 27, 2012 20:46:24

I've created another iteration of this patch. This latest one allows postponing of loading practically all scripts up until the end of the page generation cycle - or up until they're actually needed and used in the page, whichever comes first. This is very important in tuning the perceived page loading speed even further, as browsers halt page rendering and loading up until any inline script references (ie. .js files referenced in <script> blocks) are loaded. However, if we can postpone emission of all (or most) <script> elements up until to the end of the page, we're ensuring that browsers can already display most or all contents of the page, and get busy with loading the scripts only afterwards. This does not affect functionality of the page negatively, as client-side control wrapper are created anyway only at the end of the page, so most of them don't work in current Prado versions anyway until the page has finished loading - which is exactly what will happen with the new modifications, too.

The new patch also unifies handling of built-in Prado package scripts and component scripts, and introduces a new method called flushScriptFiles() to TClientScriptManager, which allows flushing of all pending script registrations into the page. This is important if you're using/building some 3rd party controls that require inline/immediate execution of some code in the page where these controls are rendered - which only works if all script dependencies are loaded at that point. This is what flushScriptFiles() takes care of (ie. that if there any outstanding script references not yet loaded in the page, they will loaded at the point said method is called, so any script code that depends on them can be executed safely at any point in the page).

Also the now-modified TClientScript is calling TClientScriptManager::flushScriptFiles() prior to rendering the script block or loading the script file referenced in it - if its newly added FlushScriptFiles property is true. This property defaults to true for compatibility reasons, so script blocks continue to work as they did previously. However, if you're not using any external functions in your TClientScript block, you should set the FlushScriptFiles property to false, so Prado can postpone loading of all the referenced script files further down the page generation cycle.

The patch also creates a separate Prado package for prototype in packages.php. This new package can be used/required by components that do not need any of the scriptaculous or Prado client-side control classes, but only prototype itself (for ex. if they're using only functions like $(), $A, $H, etc). This modification allows loading of the prototype library itself, while still postponing loading of all the other Prado base scripts - further increasing perceived page load times and speed.

I didn't have yet much time to test the new patch thoroughly, but it passed my basic tests flawlessly. Please review it whether you can find any issues with it that need to be worked around or fixed. Thanks.

[ctrlaltca]

From google...@pcforum.hu on May 28, 2012 06:49:40

I've acidentally left some debugging code in the original download. I've attached a new archive with the code in question stripped.

Attachment: prado-progressive-rendering-20120528-2.zip

[ctrlaltca]

From ctrlal...@gmail.com on June 01, 2012 11:02:32

I merged the patch on trunk/, tested it and committed as r3159