[meta] Split the core

[Hywan]

History

The core gathers several thin layers into a single library for performance reasons. Let me explain it shortly. Parsing time is not very costly compared to I/O: Accessing to a file, opening it and reading it. The core aimed at putting the minimum always needed codes into small files, in order to reduce this I/O time. Since recent PHP versions and recent I/O progresses (SSD, kernel, FS…), this is no longer necessary. We can then split the core.

Global idea

The core contains the following layers:

• Hoa\Core\Consistency,
• Hoa\Core\Data,
• Hoa\Core\Event,
• Hoa\Core\Exception,
• Hoa\Core\Parameter,
• Hoa\Core\Protocol.

We can split it into the following libraries:

• Hoa\Consistency,
• Hoa\Event,
• Hoa\Exception,
• Hoa\Protocol.

They can live outside the core without any problem. So far, though, Hoa\Core\Data and Hoa\Core\Parameter will be kept as is.

Dealing with pre-loading

Hoa\Core\Consistency, Hoa\Core\Parameter and Hoa\Core\Protocol were de-facto pre-loaded with Hoa\Core\Core. This latter is automatically pre-loaded with the autoload.files Composer's property. Whatever the split, we have to ensure this behavior.

Case of Hoa\Core\Consistency

This layer contains:

• An autoloader,
• An xcallable implementation and,
• Consistency helpers/tools accross PHP VMs (backport features to prior VMs, naming consistencies, feature consistencues etc.).

Only the latter is important at any runtime and will be loaded as a separated file with Composer. This will reduce the number of loaded code by approximately 60%.

Case of Hoa\Core\Parameter

Will not be extracted yet. Nothing to do. (see bellow)

Case of Hoa\Core\Protocol

We need to register the hoa:// protocol at any runtime. We use it everywhere. It has never hurt anyone in the whole Hoa's history. So we can load the file that is responsible to register the stream wrapper with Composer. This will reduce the number of loaded code by approximately 50%.

Steps

Here are the following steps to follow (the order is important):

• Hoa\Exception:
  • Create Hoa\Exception (from Hoa\Core\Exception),
  • Write test suites,
  • Update all libraries to use Hoa\Exception,
  • Remove Hoa\Core\Exception.
• Hoa\Event:
  • Create Hoa\Event (from Hoa\Core\Event),
  • Write test suites,
  • Update all libraries to use Hoa\Event,
  • Remove Hoa\Core\Event.
• Hoa\Protocol:
  • Create Hoa\Protocol,
  • Write test suites,
  • Update all libraries to use Hoa\Protocol,
  • Remove Hoa\Core\Protocol.
• Hoa\Consistency:
  • Create Hoa\Consistency,
  • Write test suites,
  • Update all libraries to use Hoa\Consistency,
  • Remove Hoa\Core\Consistency.

Next steps

Continue to split the core, see #97.

Here are the following steps to follow:

• Hoa\Xyl\Data:
  • Move Hoa\Core\Data to Hoa\Xyl\Data,
  • Update Hoa\View and other libraries to understand a mixed data structured instead (to remove a dependency),
  • Remove Hoa\Core\Data.
• Hoa\Parameter:
  • Create Hoa\Parameter (from Hoa\Core\Parameter),
  • Write test suites (optional so far),
  • Remove Hoa\Core\Parameter.
• Remove Hoa\Core:
  • Mark the repository as deprecated,
  • Do we empty the repository?

Update all composer.json files

Because we are dropping PHP5.4 too.

• Acl
• Bench
• Cache
• Cli
• Compiler
• Consistency
• Console
• Database
• Devtools
• Dispatcher
• Dns
• Event
• Eventsource
• Exception
• Fastcgi
• File
• Graph
• Http
• Irc
• Iterator
• Json
• Locale
• Log
• Mail
• Math
• Memory
• Mime
• Model
• Notification
• Praspel
• Promise
• Protocol
• Prototype
• Realdom
• Regex
• Registry
• Router
• Ruler
• Serialize
• Session
• Socket
• Stream
• String
• Stringbuffer
• Test
• Translate
• Tree
• Ustring
• View
• Visitor
• Websocket
• Worker
• Xml
• Xmlrpc
• Xyl
• Zformat
• Zombie
•  

[Hywan]

Thoughts @hoaproject/hoackers?

[Jir4]

And what happens to Hoa\Core\Data ? If i understand, it is kept in Hoa\Core, why ?

[Hywan]

@Jir4 Because I am not sure about what to do with this one so far.

[osaris]

That make sense, it would be very interesting to do some benchmarks before/after core splitting to check the real difference on performances (not saying that it'll decrease performances but numbers/data are interesting).

[Jir4]

@osaris 👍

[Hywan]

@osaris Also, something else must be quantified but this is really hard: The psychological impact of having a “core” for a library. This is something really important. Several people stops at this. The goal is to really continue to split. Please, see #97.

[Hywan]

I started to split Hoa\Core\Exception and Hoa\Core\Event yesterday night.

[Hywan]

Hoa\Event is almost over.
Hoa\Exception is under splitting (the repositories have been created).

[Hywan]

I am starting to update libraries with Hoa\Event and Hoa\Exception.

[Hywan]

All tests are green for Hoa\Protocol split. Now I have to write the test suite of Hoa\Protocol but the integration of the new library is fine!

[Hywan]

Everything is OK!
I am starting #97.

[Hywan]

@hoaproject/hoackers Please, I need your help!

1. What about Hoa\Zformat instead of Hoa\Parameter? Do you think this is better?
2. What to do with Hoa\Core? See the “Next steps” Section in [meta] Split the core #96 (comment) for the questions.

Thanks!

[Jir4]

1. Why ZFormat ? Meaning ?
2. I think we should delete Hoa\Core, the Core name scares many users imo, they just wanna use a specific lib but the Core dependancy seems imply that the lib need many others to work

[Hywan]

@Jir4

1. zFormat because this is the format we use to express parameters.
2. Yes, we will “remove” it, but for legacy code, it must still be present as a repository. We can mark it as deprecated and keep the code as is. Thoughts?

[Jir4]

1. Ok so 👍
2. So we keep it just to avoid a BC break and we will remove it when another BC break occurred ?

[Hywan]

@Jir4 We will remove it when we will no longer have any downloads. Until we have downloads, we must keep the repository live :-].

[CircleCode]

@Hywan the downside of this approach is that we will have downloads (at least for CI systems) so long as it will be available…
IMHO we should announce an estimated end-of-life

[Hywan]

@CircleCode Hmm… Every year, we can clean deprecated repositories? So far, we only have Hoa\String (see git.hoa-project.net). How long is enought?

[Jir4]

Several month but not years, we just have to let the time to users to update their legacy code.

[Hywan]

Success (95 tests, 800/800 methods, 0 void method, 0 skipped method, 180913 assertions)!

Hoa\Zformat has been created. I am removing hoa/core from everywhere :-).

[Hywan]

All libraries are updated. Now I am running devtools:cs to update the year.

[Hywan]

Done.

[Hywan]

Next step is to merge everything and create new snapshots for all libraries. I don't have enough time to do it well if something failed right now. Let see tonight or tomorrow!

[Hywan]

Done \o/.
Congratulations to everyone 🎆 🎉 🎈!!

[Jir4]

👍

[Hywan]

😅

[1e1]

👏
Great job!
I've just read the discussion, do you know what you earn by registering hoa:// in Composer? For projects like the official website, Rulerz, some Liip projects…

[Hywan]

@1e1 What we earn? I don't understand.

[Jir4]

I suppose that he speak about performance

[Hywan]

That's very light. Let me try a small benchmark.

[1e1]

Yeap! (Sorry)

[Hywan]

tl;dr

Here is small benchmark result when loading Hoa (because we have 1 require autoloaded file):

vs	Time (for 1000 runs)	Memory
Hoa\Core	18.084s	335408b
Hoa\Consistency	15.959s	56744b
Gain	11.751%	83.082%

Time

First script /tmp/a.sh:

for i in `seq 1 1000`; do
    php -r 'require "/usr/local/lib/Hoa/Core/Core.php";'
done

Second script /tmp/b.sh:

for i in `seq 1 1000`; do
    php -r 'require "/usr/local/lib/Hoa/Consistency/Prelude.php";'
done

Time for time:

$ time /tmp/a.sh
/tmp/a.sh  11.37s user 5.64s system 94% cpu 18.084 total

$ time /tmp/b.sh
/tmp/b.sh  9.56s user 5.07s system 91% cpu 15.959 total

That's an improvement of 11.751% at loading time.

Memory

First script /tmp/a.php:

<?php

$a = memory_get_usage();

require '/usr/local/lib/Hoa/Core/Core.php';

var_dump(memory_get_usage() - $a);

Second script /tmp/b.php:

<?php

$a = memory_get_usage();

require '/usr/local/lib/Hoa/Consistency/Prelude.php';

var_dump(memory_get_usage() - $a);

Time to run:

$ php /tmp/a.php
int(335408)

$ php /tmp/b.php
int(56744)

That's an improvement of 278664b, be 272Kb, be 83.082% at initialization per process per run.