[Discussion] Consider using some (MVC) framework for the web-interface and API?

[nickygerritsen]

While talking to @tuupke about some DOMjudge ideas (mainly for the still WIP online judge rework thingy of us), we ended up in a discussion about why we are not using some framework for the web interface and API.

I partially know the answer myself (no experience and it grew this way), but I think we should start considering it. Last weekend during the hackathon our great friend Jan already noticed the current code is mixing UI and controller code like it's nothing. This worked for simple pages, but we are getting more and more complex pages, like the testcase page. Jan started working on trying to get Twig integrated for that page, but perhaps we should move it a step further and use some framework like Symfony or Laravel (or another one...).

I think the web interface and API of DOMjudge are not too difficult: I think the difficult part is in the judgedaemon and related things.

This issue is to discuss whether we even see this as an option :).

To start of some discussion, let me add some advantages and disadvantages which I see.

Advantages:

• We can use the strong base of a framework to do a lot of stuff. For example a framework has a coupled templating engine, database wrapper, etc. lib.database.php served us well for a long time but I think for most things some ORM would make more sense.
• It is easier for (external) developers to add code, because we use some (well-known) setup.
• Separating UI and logic is a thing we already want to do and a framework will almost force us to do so automatically.
• Doing something like the API we currently provide is almost trivial in a good framework.
• So is doing authentication.
• We can lift upon (security) updates from the framework.

Disadvantages:

• There are some queries for which an ORM is maybe hard. I can image some of the scoreboard things to be hard. However, normally we could bypass the ORM for some specific things. Or for example Doctrine ORM allows you to write ORM-style selectors which are a close mapping to an SQL query.
• It will be quite some work to do this: it won't be something we can do in a week. We need to rethink and/or rewrite a lot of our code.
• During the "transition period" we will add features using the current structure which we need to re-do when the transition is complete.

Some might think the performance will drop using a framework. However, I don't believe this is the case for a good framework. Frameworks nowadays have a lot of mechanisms to have no or only a little performance impact and I think that using a framework will even speed up some things, because the framework can already have good support for caching stuff etc.

What do other people think about this idea?

I don't want to really discuss which framework etc. yet, as I think we first need to decide whether we want to use it at all.

[eldering]

First of all, I don't have any experience with PHP frameworks nor with templating, so writing from that perspective.

I think indeed the main point in deciding whether to use a framework (and which one) is whether it makes the code easier to develop/maintain, both for current developers and new contributors. I think the current code for webpages is not always clear, especially since it mixes logic with presentation. Clearly a template system will help there, but most of the logic is fairly straightforward and understandable.

Secondly there are some home-grown solutions to abstract away e.g. form building, DB relations and data validation, etc. I think these are the most difficult to understand from the point of view of a new contributor (and sometimes even to me ;-) On the other hand, a PHP framework also adds an extra layer of abstraction, which even though standard, has to be learned too. Therefore, I think the most important questions here are:

• Is it possible in the framework to express (nearly) all logic that we currently use?
• How easy is it to write customized code for things that don't fit inside the framework?
• How much unncessary abstraction does the framework add?
• How many extra dependencies does the framework add, i.e. does it run on latest Debian/Ubuntu stable (LTS) releases?

So I think a framework could be a good choice, but it may depend a lot on the details of the framework (again, I don't know these frameworks, so cannot judge this). In particular, I'd be interested to see how some of the more complex DB queries could be handled within a framework and whether on these the performance is still acceptable.

So my proposal would be to investigate which framework(s) would suit us best and then choose one or two pages with complicated logic and try implement these inside the framework.

[nickygerritsen]

I think indeed the main point in deciding whether to use a framework (and which one) is whether it makes the code easier to develop/maintain, both for current developers and new contributors. I think the current code for webpages is not always clear, especially since it mixes logic with presentation. Clearly a template system will help there, but most of the logic is fairly straightforward and understandable.

True about easy logic. I indeed think separating those will make sense for new devs.

Secondly there are some home-grown solutions to abstract away e.g. form building, DB relations and data validation, etc. I think these are the most difficult to understand from the point of view of a new contributor (and sometimes even to me ;-) On the other hand, a PHP framework also adds an extra layer of abstraction, which even though standard, has to be learned too.

True, but then at least you can Google for questions instead of needing to read the contents of lib.database.php or edit.php ;-).

Therefore, I think the most important questions here are:

• Is it possible in the framework to express (nearly) all logic that we currently use?

I think we should somehow find out what the hard parts of our logic are so we can try to understand how to implement this in a framework.

• How easy is it to write customized code for things that don't fit inside the framework?

This of course depends on the framework. Also, we need to know what doesn't fit in it.

• How much unncessary abstraction does the framework add?

I think this is something we can only find out when trying frameworks. Also sometimes it is not too much of a problem if it adds abstraction, as long as it makes some sense and doesn't impact performance, etc.

• How many extra dependencies does the framework add, i.e. does it run on latest Debian/Ubuntu stable (LTS) releases?

Yup we at least need to check this.

So I think a framework could be a good choice, but it may depend a lot on the details of the framework (again, I don't know these frameworks, so cannot judge this). In particular, I'd be interested to see how some of the more complex DB queries could be handled within a framework and whether on these the performance is still acceptable.
So my proposal would be to investigate which framework(s) would suit us best and then choose one or two pages with complicated logic and try implement these inside the framework.

The only framework I work with regularly is Symfony. For this I can already answer some questions:

• Version 2.8 (latest LTS) requires PHP 5.3.9+ and JSON and ctype need to be enabled, according to http://symfony.com/doc/2.8/reference/requirements.html. 3.0 (latest normal release) required PHP 5.5.9+.
• Symfony is in its core a list of useful smaller projects bundled together, which can help in not using unneeded abstraction. They offer a "default set" of stuff which normally makes sense, but they also have a REST-API set of stuff for example.
• They use Twig for rendering and Doctrine as an ORM. Most simple-ish queries are easy to do in Doctrine and you can write something similar to SQL for complex queries. More on database-stuff below.

About the DB queries: because the database ORM for most frameworks is pretty self-contained, we could first try to do one or more complex queries in the ORM. I would be willing to try this for Doctrine, but then we should maybe figure out some complex queries :-).

I am curious whether other people have experience with other frameworks btw. Or bad experiences with Symfony.

[tuupke]

Since I was mentioned in the original question I feel I should also respond. Even though I have never directly contributed to dj.

For me the issue with starting to develop on dj is the (perceived) high level of knowledge required to start. Since many features are split over multiple files and are loaded in a (sometimes) odd order. Personally, deciding what needs to be changed where in order to implement a feature while making sure nothing else breaks is a huge task and nigh impossible, in my opinion.

Having a framework which (semi) forces you to work using a certain design pattern alleviates this entire issue. @nickygerritsen basically summed it up pretty good. Most frameworks provide some database abstraction using an ORM, provides some form of templating, and maybe even more important: gives access to libraries with implementations. So new features do not necessarily have to be implemented since an "off-the-shelf" solution is available. Or the library at least has the basics covered.

On the choice of frameworks. Personally I have much (positive) experience with Laravel & Lumen and bad experiences with Doctrine (the ORM of Symfony). Mainly since DQL (the SQL dialect used in Doctrine) has no support for some common features of SQL like unions and excepts. But many, many frameworks exist some provide high levels of abstractions and force you to work in some manner (like Laravel and Symfony) while others are more "flexible" (like Slim) but all provide basically the same feature set.

• How many extra dependencies does the framework add, i.e. does it run on latest Debian/Ubuntu stable (LTS) releases?

That is one of the exact issues these frameworks have to deal with, therefor new versions of these frameworks always require php versions which are in the (latest) stable versions of common server operating systems.

Another options could be to self-implement a framework, but then in a few years dj would end up in the situation where it is now.

All in all I think there are way more advantages then disadvantages. it would lead to a more maintainable project. While also being accessable for new contributors since they can use a structure and default tooling (like composer) which they are used to.

[nickygerritsen]

What @tuupke says is quite true I think; he often texts me about how to do stuff in DOMjudge, because it currently is not really in a logical place.

I myself have no Laravel / Lumen experience and have not had problems with DQL yet. But I haven't tried doing a UNION or EXCEPT query in it. I did do a little grep on the code and we do not use UNION / EXCEPT currently. I'm also guessing (not sure because I'm no SQL pro) that those two are not ideal to use in MySQL as it will probably use temporary tables for them.

Anyway, I'm not opposed to any framework currently but I do think it would make sense that we use one that at least someone in the team has experience with.

[jbransen]

Yes, this is a good idea. I've been thinking about this for a while and I believe with the introduction of the API, the codebase has reached the point where using some abstractions is really necessary to keep the code maintainable.

I think there are two important improvements to make:

1. Using some ORM to abstract over all database queries
2. Use some templating system to separate the HTML from the PHP code

There are frameworks that can do both (and much more), and I am not sure if that is a good idea. As @eldering points out framework can cause unnecessary abstractions. I would suggest to tackle both points (sort-of) separately, and drop the idea of using a big framework to do this all at once.

The main motivation is that the transition can then be done easily. When introducting an ORM the database queries can slowly be rewritten over time, and the same holds for the front end pages. Of course the choice for the concrete frameworks/design patterns for this is then still open.

[nickygerritsen]

I think for the first part this can be a valid approach, but if we only introduce a templating engine for the views, we still have a lot of PHP code that are now put in "some" lib-files, edit.php for (almost) all form submission handling, etc.

For me this feels as almost a big a mess as the mixing of the view-related code and PHP. Using templates only will not really solve that part I guess. It is nice that then the real view-related code is separate, but in I think this is only a part of the solution.

However, we might also tackle that point (a lot of shared code everywhere) separately from the first two, although I do think that in the end you then again end up with either something that is very similar to some framework but not understandable for a "DOMjudge noob" or actually a real framework.
Not sure though.

[nickygerritsen]

After thinking a bit more about this in bed (....), I'd like to change my previous opinion:

Although first using a ORM for the database queries solves one part (not needing to do SELECT * FROM... everywhere), I think it does not solve my (and I think also @tuupke's) main concern that the code will stay "messy" and "homegrown" if we use this approach. For me this is a bigger problem than having SQL queries in the code and I think @tuupke also means this: if we use some "standard" framework it is way easier for others (and us too) to "get into" DOMjudge and add features.

[eldering]

My feeling is that to get this discussion forward, we should start looking into choosing a specific PHP framework and try to implement things to see how it would work.

As already mentioned (not sure in this thread) I think we should choose a framework that at least some of the active developers are familiar with. Further, it would be good if the framework is modular so that we can start rewriting code for templatiing, ORM, and other things in separate phases. It sounds to me that Symfony does fit these requirements, while the objection of not supporting UNION and EXCEPT is not relevant for the current code. Also, Jan started reimplementing the testcase page in Twig. Any (new) arguments for/against using Symfony to start an implementation test?

[nickygerritsen]

To add something here: I tried a really small test last sunday where I imported our database scheme in Doctrine (Symfony's ORM) and that seemed to work quite nice.

Within a few moments I could write

{% for user in users %}
  {{ user.username}} {% if user.team is not null %}{{user.team.name}}{% endif %}
  {% for role in user.roles %}
    {{ role.role }}
  {% endfor %}
{% endfor %}

Without writing a query :-).

Of course users are quite simple in our database and stuff like the scoreboard will be a bit more complex.

[nickygerritsen]

Btw perhaps we should make a list of a few things we would like to see how they work in a framework, so we can focus on that. For example:

• A simple page for managing an entity, i.e. a user and/or team
• A more complex page?
• An API action
• Some complex query-thingy?
• etc.

I think we could start with just creating a separate repository for testing purposes where we do some of these things.

[tuupke]

If there is a plan to start doing this, I'd love to start helping out, as stated before I've had many things I wanted to implement (and have implemented) in dj, but never felt it was good enough (or well tested enough) to submit a pr.

For the people that are interested, this page contains an extensive comparison of many popular frameworks.

[eldering]

Ok, I've created a new branch framework-rewrite on top of current master. Feel free to experiment in there!

I also invited @tuupke as a member, so you can commit directly to the repository.

[nickygerritsen]

OK I'll try to see if I can do some experiments in Symfony, but maybe @tuupke wants to look at (an)other framework(s)? Maybe it makes sense to make a different branch for each, but I guess I can do that myself.

[tuupke]

@eldering Thanks!

@nickygerritsen Mhh, what do you propose to do experiments on? What do you want to find out specifically?

[nickygerritsen]

That's why my list was there. I gues Jaap or someone could extend the list?