Add explanation about implicit indexes

[greg0ire]

Q	A
Type	docs
BC Break	no
Fixed issues	n/a

Summary

I'm unsure about the target branch: I wouldn't want this to block the next 2.10 release, which should be the last.
Also, I did not document the known issue about not supporting usage on table created by another tool that does not create implicit indexes where DBAL does not itself create some, because I think it will be fixed soonish, but tell me if you feel it would be worth mentioning

Also, I have named the parent directory explanation in to the divio naming, but I am also noticing there is a design (https://github.com/doctrine/dbal/tree/2.11.x/docs/design)
directory up one level that contains md files that make the docs website strangely (one can see these documents pop up in the search, but lead to 404s). Maybe that directory should be merged with explanation and its contents be merged with this one? Or maybe is it more of a cookbook? Anyway it would be great to decide beforehand if explanation is right name, or if we should name it design. I did not have a look at packages to see if either name might be already used.

And finally, does anyone know how I can build the docs locally?

[morozov]

While it's a really well-written piece of explanation, I'm not sure how it's expected to help the DBAL users. It could be an explanation of how DBAL abstracts out the differences between platforms but it doesn't do that.

As far as I understand, DBAL will always create an explicit index that corresponds to the FK regardless of whether the underlying platform automatically creates it, so it's always covered.

I may be wrong about the facts, but the structure that I'd expect to see is the following:

1. Different databases handle FK indexes differently.
2. Our intent is to make sure that the indexes are always created (explicitly or implicitly, this is the implementation detail that the developer shouldn't be bothered about because of DBAL).
3. Internally, if the platform creates explicit indices, this is what will happen, if it creates implicit indexes, then this, and if it doesn't, then this.

P.S.: 2.11.x looks good as the target branch.

[greg0ire]

While it's a really well-written piece of explanation, I'm not sure how it's expected to help the DBAL users. It could be an explanation of how DBAL abstracts out the differences between platforms but it doesn't do that.

Good point about reducing the differences, I will try to add something about that

As far as I understand, DBAL will always create an explicit index that corresponds to the FK regardless of whether the underlying platform automatically creates it, so it's always covered.

I'm not sure about that. Consider the following piece of code:

dbal/lib/Doctrine/DBAL/Schema/Table.php

Lines 570 to 574 in f97ee94

	foreach ($this->_indexes as $existingIndex) {
	if ($indexCandidate->isFullfilledBy($existingIndex)) {
	return;
	}
	}

When you introspect a table created with MySQL, won't the RDBMS-defined index be listed in $this->_indexes, and fulfill the index candidate? If not, it means we would have a duplicate DBAL-defined index for nothing.

I may be wrong about the facts, but the structure that I'd expect to see is the following:
Different databases handle FK indexes differently.
Our intent is to make sure that the indexes are always created (explicitly or implicitly, this is the implementation detail that the developer shouldn't be bothered about because of DBAL).
Internally, if the platform creates explicit indices, this is what will happen, if it creates implicit indexes, then this, and if it doesn't, then this.

Ok, I will try to rework this to match that structure.

[greg0ire]

When you introspect a table created with MySQL, won't the RDBMS-defined index be listed in $this->_indexes, and fulfill the index candidate? If not, it means we would have a duplicate DBAL-defined index for nothing.

@guilhermeblanco hopefully you know about this? It looks like you are the most familiar with this feature.

[greg0ire]

@morozov I overlooked your main question, let me address it.

I'm not sure how it's expected to help the DBAL users. It could be an explanation of how DBAL abstracts out the differences between platforms but it doesn't do that.

The goal I had in mind for this document was to answer the question "What are these weirdly-named indexes I can see appear when running migrations, and I did not ask for?". It's targeted at users, but also at us maintainers (it would have helped us understand one of the SQLite bugs faster).

[morozov]

What are these weirdly-named indexes I can see appear when running migrations, and I did not ask for?

This looks like the right question to me. Are those the implicit indexes in the title of the article or they are DBAL-defined? The other question is, will the user see them in the migration script or in the DB eventually? They will see DBAL-defined ones in the script but will only see the implicit ones in the deployed schema (e.g. on MySQL).

I apologize if it sounds like a criticism but if that is the primary question you want to answer, maybe the following structure could be used?

1. Why am I seeing those indices? They are needed for FKs to perform well.
2. Some platforms will take care of them and generate automatically. Then DBAL will not generate them in migration scripts and will rely on the platform. Maybe a code example (again, if that' the case, I don't know).
3. Some platforms don't care. Then DBAL steps in: example.

I admit that I may misunderstand the purpose of the article. This is by no means an attempt to direct here.

[greg0ire]

What are these weirdly-named indexes I can see appear when running migrations, and I did not ask for?
This looks like the right question to me. Are those the implicit indexes in the title of the article or they are DBAL-defined?

Weirdly-named indexes are DBAL-defined.

You can see here the name that MariaDB picks for an implicit index: https://dbfiddle.uk/?rdbms=mariadb_10.5&fiddle=11d4234c37b192c46cd6566efdb9799c (it picks trackartist, so… less weird?)

I admit that I may misunderstand the purpose of the article. This is by no means an attempt to direct here.

There are several purposes, the main purpose is to answer that first question, the secondary purposes would be to serve as a reference when a maintainer such as you and I see these indexes being talked about in bug reports, and don't remember what they are for, and also to explain the decisions that lead to this, so that people can better challenge these decisions if necessary. You did so a little IIRC, when you said you thought this might be a disservice to the user.

[greg0ire]

@morozov I had a doubt about how this works so I tried the demo symfony application (symfony new --demo my_project) with a mariadb server started with DB=mariadb.docker IMAGE=mariadb:10.3 bash ./tests/travis/docker-run-mysql-or-mariadb.sh
the connection string that can be changed in .env is mysql://root@127.0.0.1:33306/doctrine_tests, and sure enough, it also creates the DBAL-defined indexes for MariaDB. Using SHOW INDEXES FROM symfony_demopost shows the DBAL-defined index. If I comment out the code block responsible for that index, and recreate the schema, I still have 2 indexes, but this time the index is named FK_… (exactly like the foreign key) so this must be an RDBMS-defined index.

This does not work like I thought it would: the DBAL always creates that index regardless of whether the platform creates implicit indexes or not, and that makes sense since it can't read the table and detect that beforehand since it does not exist yet. Instead the DBAL leverages the feature that I describe in the last paragraph of the explanation:

Some RDBMS such as MySQL do something similar and can also drop implicit indexes that are fulfilled by user-defined indexes.

I will have to rework this but one thing is for sure, this explanation will have to be reviewed by people familiar with that feature otherwise misconceptions might slip in.

[morozov]

and can also drop implicit indexes

This still looks like a misconception to me. In my understanding, MySQL will ensure that the FK is covered by an index. If the index isn't provided by the user, it will create its own. I.e. it doesn't drop anything.

[greg0ire]

If you take https://dbfiddle.uk/?rdbms=mariadb_10.5&fiddle=11d4234c37b192c46cd6566efdb9799c as an example, you can see an RDBMS-defined index.

Now if you add an index that fulfills the same needs, this is what you obtain:

https://dbfiddle.uk/?rdbms=mariadb_10.5&fiddle=d40ff6c0242591ff4163b96ecda23f3b

To me, it looks like MariaDB dropped the existing index

[greg0ire]

The MariaDB docs I linked to are not super clear about this:

If MariaDB automatically creates an index for the foreign key (because it does not exist and is not explicitly created), its name will be index_name

They make it sound like MariaDB decides to add an index because there is none, but don't say it will be dropped if you add one.

The Mysql docs I linked to are a bit more clear:

MySQL requires indexes on foreign keys and referenced keys so that foreign key checks can be fast and not require a table scan. In the referencing table, there must be an index where the foreign key columns are listed as the first columns in the same order. Such an index is created on the referencing table automatically if it does not exist. This index might be silently dropped later if you create another index that can be used to enforce the foreign key constraint. index_name, if given, is used as described previously.

[morozov]

Thanks for the details, @greg0ire. I misinterpreted the earlier statement. I believe the same logic that is explicitly defined for MySQL is also true for MariaDB.

Some RDBMS such as MySQL do something similar and can also drop implicit indexes that are fulfilled by user-defined indexes.

Probably this confusion could be avoided if we put it as “can drop an existing implicit index once it's fulfilled by a newly created user-defined index”.

[greg0ire]

@morozov I just pushed a reworked version, the only thing I did not fix is the title, which, as you pointed out, is ambiguous: are we talking about DBAL-defined indexes, or about RDBMS-defined indexes? The document covers both, so I think it can stay as is.

[morozov]

Structure- and content-wise, looks good. A couple of minor notices.

[morozov]

🚢

[SenseException]

but I am also notice there is a design (https://github.com/doctrine/dbal/tree/2.11.x/docs/design) directory up one level that contains md files that make the docs website strangely

Azure won't be much of an issue anymore starting with 3.0. The documentation already has a sharding section and because the last changes of these two files in design seem to be 3 years ago, I would suggest to delete the directory in a PR.