[wip] 3.0 - Implement connection prefixes (#2666) #3843

Closed
wants to merge 49 commits into
from

Projects

None yet

5 participants

@HavokInspiration
Member

This PR aims to implements connection prefixes (#2666).

Since it's my first PR ever on an open source project, feel free to tell me if I did anything wrong.
I also have a some questions regarding "Cakey" way to do things :

1/ Should the Connection::fullTableName() method throws an exception if the $name parameter is neither a string nor an array ?
2/ Are there any missing tests ? (ie. should I write test for the Query::from() and Query::join() methods as well as the Collection::describe() ? I tried for the last one but was not successful in having correct results. I was hopping a core contributor could help me write this test)

HavokInspiration added some commits Jun 29, 2014
@HavokInspiration HavokInspiration Implements connection prefixes 4572e82
@HavokInspiration HavokInspiration Tests for the Connection::fullTableName method 606dea8
@HavokInspiration HavokInspiration Clean up doc block 66bd161
@HavokInspiration HavokInspiration More clean up
06417ab
@lorenzo lorenzo and 1 other commented on an outdated diff Jun 30, 2014
src/Database/Connection.php
+ *
+ * Get the full table(s) name with prefix, if any.
+ *
+ * @param mixed $name The name of the table or an array of table names
+ *
+ * @return mixed Full table name or array of table names
+ *
+ */
+ public function fullTableName($name) {
+ if (empty($this->_config["prefix"])) {
+ return $name;
+ }
+
+ $prefix = $this->_config["prefix"];
+ if (is_string($name)) {
+ $name = $prefix . $name;
@lorenzo
lorenzo Jun 30, 2014 Member

What happens if the table is object? For example another query

@markstory
markstory Jun 30, 2014 Member

I would also consider the case where a table name ends up in this function multiple times. The old connection prefix code had problems with tables being double prefixed a few times.

@markstory markstory added this to the 3.0.0 milestone Jun 30, 2014
@markstory markstory commented on an outdated diff Jun 30, 2014
src/Database/Connection.php
@@ -189,6 +189,34 @@ public function isConnected() {
}
/**
+ *
+ * Get the full table(s) name with prefix, if any.
+ *
+ * @param mixed $name The name of the table or an array of table names
+ *
+ * @return mixed Full table name or array of table names
+ *
+ */
+ public function fullTableName($name) {
+ if (empty($this->_config["prefix"])) {
@markstory
markstory Jun 30, 2014 Member

You might want to use !isset() in case some joker sets their prefix to '0'.

HavokInspiration added some commits Jul 1, 2014
@HavokInspiration HavokInspiration Tests if _connection is an instance of Connection
This should fixe the Travis build failure
ad82cce
@HavokInspiration HavokInspiration Changed to an isset in case the prefix is 0 b8e1217
@HavokInspiration HavokInspiration Merge remote-tracking branch 'upstream/3.0' into 3.0
5b05b8c
@HavokInspiration HavokInspiration Tests if _connection is an instance of Connection
This should fixe the Travis build failure
9db1d33
@HavokInspiration HavokInspiration Changed to an isset in case the prefix is 0
4817056
@HavokInspiration HavokInspiration Merge remote-tracking branch 'origin/3.0-wip-connection-prefixes' int…
…o 3.0-wip-connection-prefixes
c51e2f4
@HavokInspiration HavokInspiration Only apply prefix if $tableName is a string 8bbd486
@HavokInspiration HavokInspiration Provides consistency with what the variable represents 9120396
@HavokInspiration HavokInspiration DocBlock b750668
@HavokInspiration HavokInspiration Update test case with subquery case
75edd8b
@HavokInspiration
Member

@lorenzo Indeed, if $tableName is the loop happened to be a Query object, everything broke.

@markstory While I understand the issue concerning double prefixed table names, I'm not quite sure how to proceed. I could check if the table name doesn't start with the prefix but if one of the table name (not prefixed) starts with the prefix defined, then the name would not be prefixed (I don't know if I am being really clear...)
The other solution would be to have an array that stores all prefixed tables but the problem described above would still occur...

@markstory
Member

@HavokInspiration One option would be instead of manipulating strings for table names, table names could be wrapped in a TableNameExpression that appends the prefix name when it is converted into SQL. Having a wrapper object for prefixed table names would prevent double prefixing and should work well with the existing query builder internals.

@HavokInspiration
Member

Just pushed my first draft on the TableNameExpression as you suggested @markstory. It probably needs some polishing.

Currently I have failing tests in \Cake\ORM (well, errors actually).
It seems that one of the property of the Expression (both name and value) is ending in some queries that are generated making the SQL queries invalid. I'll take a deeper look into it later.

I just wanted to submit this so you could review my work so far.

@markstory markstory and 1 other commented on an outdated diff Jul 8, 2014
src/Database/Connection.php
+ *
+ * @param string|array|\Cake\ORM\Query $names The names of the tables
+ *
+ * @see \Cake\Database\Expression\TableNameExpression
+ * @return string|array|\Cake\ORM\Query Full tables names
+ *
+ */
+ public function fullTableName($names, $type = "from") {
+ $prefix = "";
+
+ if (isset($this->_config["prefix"]) && $this->_config["prefix"] !== "") {
+ $prefix = $this->_config["prefix"];
+ }
+
+ if (is_array($names) && !empty($names)) {
+ foreach ($names as $alias => &$tableName) {
@markstory
markstory Jul 8, 2014 Member

I wouldn't use a reference here personally.

@HavokInspiration
HavokInspiration Jul 8, 2014 Member

I'll remove it. It's legacy, I used it as a reference on the first version but I changed it on my last commits but forgot this.

@markstory markstory and 2 others commented on an outdated diff Jul 8, 2014
src/Database/Expression/TableNameExpression.php
+ * @param string $name
+ * @return void
+ */
+ public function setName($name) {
+ $this->_name = $name;
+ }
+
+/**
+ * Sets the prefix for the table name of this expression
+ *
+ * @param string $prefix
+ * @return void
+ */
+ public function setPrefix($prefix) {
+ $this->_prefix = $prefix;
+ }
@markstory
markstory Jul 8, 2014 Member

We normally avoid lots of setter methods. Most of the expression objects are value objects that don't need to be mutable.

@dereuromark
dereuromark Jul 8, 2014 Member

Also, wouldn't it then be better to chain them by returning "$this" instead of void?

@HavokInspiration
HavokInspiration Jul 8, 2014 Member

Ok, I'll remove them.

@markstory markstory commented on an outdated diff Jul 8, 2014
src/Database/Expression/TableNameExpression.php
+ $this->setPrefix($prefix);
+ $this->setAlias($alias);
+ $this->setType($type);
+ }
+
+/**
+ * Converts the expression into a SQL string fragment.
+ *
+ * @param \Cake\Database\ValueBinder $generator Placeholder generator object
+ * @return string
+ */
+ public function sql(ValueBinder $generator) {
+ $sql = "";
+ $quote = false;
+
+ if (is_object($this->_driver) && method_exists($this->_driver, 'autoQuoting')) {
@markstory
markstory Jul 8, 2014 Member

How does the _driver property get set? I don't see it in the constructor, or runtime code.

@markstory markstory and 1 other commented on an outdated diff Jul 8, 2014
src/Database/IdentifierQuoter.php
@@ -131,6 +139,27 @@ protected function _basicQuoter($part) {
}
/**
+ *
+ * @todo comment
+ *
+ */
+ protected function _quoteFroms($froms) {
+ $result = [];
+
+ if (!empty($froms)) {
+ foreach ($froms as $alias => $value) {
+ if ($value instanceof TableNameExpression) {
+ $value->setDriver($this->_driver);
@markstory
markstory Jul 8, 2014 Member

It is a bit odd that this is the only expression object that gets a driver set to it.

@HavokInspiration
HavokInspiration Jul 8, 2014 Member

Well, it's actually the only solution I found so far as when quoting appends, $value is still an Expression. I could __toString() it in order to quote it at this moment but there's no instance of the ValueBinder.
I guess I could mutate the name property in the Expression instance by calling the quoteIdentifier method on it (only if the name is a string) like :

if ($value instanceof TableNameExpression) {
    $tableName = $value->getName();
    if (is_string($tableName)) {
        $quoted = $this->_driver->quoteIdentifier($tableName);
        $value->setName($quoted);
    }
}

It would then prevent each TableNameExpression to hold an instance of the driver.

I'll give it a shot.

@markstory markstory and 1 other commented on an outdated diff Jul 8, 2014
src/Database/IdentifierQuoter.php
@@ -150,6 +180,10 @@ protected function _quoteJoins($joins) {
$value['table'] = $this->_driver->quoteIdentifier($value['table']);
@markstory
markstory Jul 8, 2014 Member

Wouldn't this need to change as well?

@HavokInspiration
HavokInspiration Jul 8, 2014 Member

Yes, I missed it.

@markstory markstory commented on an outdated diff Jul 8, 2014
tests/TestCase/Database/ConnectionTest.php
+ // $tableNames = ["Posts" => "posts", "Users" => "users"];
+ // $expected = ["Posts" => "prefix_posts", "Users" => "prefix_users"];
+ // $subQuery = ["sub" => $this->connection->newQuery()->select('1 + 1')];
+
+ // $fullTableName = $connectionNoPrefix->fullTableName($tableName);
+ // $this->assertEquals($fullTableName, $tableName);
+
+ // $fullTableName = $connectionPrefix->fullTableName($tableName);
+ // $this->assertEquals($fullTableName, $config["prefix"] . $tableName);
+
+ // $fullTableNames = $connectionPrefix->fullTableName($tableNames);
+ // $this->assertSame($expected, $fullTableNames);
+
+ // $fullTableNameSubQuery = $connectionPrefix->fullTableName($subQuery);
+ // $this->assertSame($subQuery, $fullTableNameSubQuery);
+ // }
@markstory
markstory Jul 8, 2014 Member

Test is commented out?

HavokInspiration added some commits Jul 8, 2014
@HavokInspiration HavokInspiration Remove unnecessary setters 241c935
@HavokInspiration HavokInspiration Remove reference b333484
@HavokInspiration HavokInspiration Change the way the TableNameExpression is quoted when autoQuoting is …
…set to true

This changes prevent a instance of the driver to be stored in the TableNameExpression instance. It simplifies the TableNameExpression and restore the way the table name alias are binded in the query.
ae03f1e
@HavokInspiration HavokInspiration An expression should not be prefixed c1999ae
@HavokInspiration HavokInspiration Merge remote-tracking branch 'upstream/3.0' into 3.0
36fd695
@HavokInspiration HavokInspiration changed the title from 3.0 - Implement connection prefixes (#2666) to [wip] 3.0 - Implement connection prefixes (#2666) Jul 8, 2014
@HavokInspiration
Member

@markstory I pushed some changes based on your comments.
All tests should pass (they passed before I pushed).
Next step is to write some tests for the feature. I might need some help on those.

@markstory markstory and 1 other commented on an outdated diff Jul 8, 2014
src/Database/Connection.php
+ */
+ public function fullTableName($names, $type = "from") {
+ $prefix = "";
+
+ if (isset($this->_config["prefix"]) && $this->_config["prefix"] !== "") {
+ $prefix = $this->_config["prefix"];
+ }
+
+ if (is_array($names) && !empty($names)) {
+ foreach ($names as $alias => $tableName) {
+ if (is_string($tableName) || $tableName instanceof Query || $tableName instanceof QueryExpression) {
+ $names[$alias] = new TableNameExpression($tableName, $prefix, $type, $alias);
+ }
+ }
+ } else {
+ $names = new TableNameExpression($names, $prefix, $type);
@markstory
markstory Jul 8, 2014 Member

Will this do the right thing if $names is an empty array?

@HavokInspiration
HavokInspiration Jul 9, 2014 Member

Well no, it shouldn't go through this. I will get this fixed.

@markstory markstory and 2 others commented on an outdated diff Jul 8, 2014
src/Database/Expression/TableNameExpression.php
+
+/**
+ * Gets the table name this expression represents
+ *
+ * @return string Table name this expression represents
+ */
+ public function getName() {
+ return $this->_name;
+ }
+
+/**
+ * Change the $_quoted property that to tell that the $_name property was quoted
+ *
+ * @return void
+ */
+ public function isQuoted() {
@markstory
markstory Jul 8, 2014 Member

This sounds like a method to check not a method to mutate the object. Perhaps enableQuoting() or setQuoted()?

@josegonzalez
josegonzalez Jul 8, 2014 Member

The method should probably also come after the constructor.

@HavokInspiration
HavokInspiration Jul 9, 2014 Member

Good point.
I will probably go for setQuoted to keep consistency.

@markstory markstory and 1 other commented on an outdated diff Jul 8, 2014
src/Database/IdentifierQuoter.php
+ if (!empty($froms)) {
+ foreach ($froms as $alias => $value) {
+ if ($value instanceof TableNameExpression) {
+ $tableName = $value->getName();
+ if (is_string($tableName)) {
+ $quoted = $this->_driver->quoteIdentifier($tableName);
+ $value->setName($quoted);
+ $value->isQuoted();
+ }
+ } else {
+ $value = !is_string($value) ? $value : $this->_driver->quoteIdentifier($value);
+ }
+
+ $alias = is_numeric($alias) ? $alias : $this->_driver->quoteIdentifier($alias);
+
+ $result[$alias] = $value;
@markstory
markstory Jul 8, 2014 Member

Is there a way we can do this with fewer than 5 levels of indentation?

@HavokInspiration
HavokInspiration Jul 9, 2014 Member

I guess it could be done through a method which could even be used in _quoteJoins to keep it DRY.

@markstory markstory commented on the diff Jul 8, 2014
src/Database/Query.php
@@ -374,6 +374,10 @@ public function from($tables = [], $overwrite = false) {
$tables = [$tables];
}
+ if ($this->_connection instanceof \Cake\Database\Connection) {
@markstory
markstory Jul 8, 2014 Member

Why is this here?

@HavokInspiration
HavokInspiration Jul 9, 2014 Member

Some tests (in \Cake\ORM\ if I remember correctly) creates an instance of \Cake\ORM\Query with a null connection parameter. Errors were raised by those tests since it was calling a method from a non-object. So I had to make sure that the _connection property was what it should be.

@markstory
markstory Jul 9, 2014 Member

We should probably fix those tests, they are probably wrong in other ways as well.

@dereuromark
dereuromark Jul 9, 2014 Member

Yep, the second argument is already fixed in https://github.com/cakephp/cakephp/pull/3802/files#diff-697d798d5d5411ad5dce707736c9cfb5R105 - as it now has to be of type Table.
But the first one still often can be null.

HavokInspiration added some commits Jul 11, 2014
@HavokInspiration HavokInspiration Changed method name to setQuoted to reflects its purpose bb04283
@HavokInspiration HavokInspiration Clean up TableNameExpression constructor
Removed parameters that were not used anymore because of changes made in past commits.
25d2e04
@HavokInspiration HavokInspiration Clean up namespace calls a9f9646
@HavokInspiration HavokInspiration Reduces indentation by creating a function
This also helps keep the code DRY
36a5685
@HavokInspiration HavokInspiration Change param and return variable type
According to \Cake\Database\Query::from() and \Cake\Database\Query::join(), only string, array and ExpressionInterface type can be passed as table name. These changes aims to follow this.
0778cdd
@HavokInspiration HavokInspiration DocBlock corrections c2e5a66
@HavokInspiration HavokInspiration Update \Cake\Database\Connection::fullTableName() tests 9c0dd42
@HavokInspiration HavokInspiration Deals with the case where $names is an empty array d29296c
@HavokInspiration HavokInspiration Update test ed09dd1
@HavokInspiration HavokInspiration Merge remote-tracking branch 'upstream/3.0' into 3.0
e29158b
@HavokInspiration
Member

Hm... Tests are failing on Travis but when I run them on my local workspace, all tests pass... I don't understand why.

Edit : OK, I understand why... they are all failing except with SQLite.

Edit 2 : From what I gathered in other PR, it seems to be an "general" issue.

HavokInspiration added some commits Jul 11, 2014
@HavokInspiration HavokInspiration Add tests for TableNameExpression
0202f4c
@HavokInspiration HavokInspiration Update TableNameExpression tests
Removes the hardcoded quoters
2800963
@HavokInspiration HavokInspiration Merge remote-tracking branch 'upstream/3.0' into 3.0
f11890b
@HavokInspiration HavokInspiration Fix CS
63d504e
@HavokInspiration HavokInspiration Merge remote-tracking branch 'upstream/3.0' into 3.0-wip-connection-p…
…refixes
ea66a59
@HavokInspiration HavokInspiration Updates the new join method to use the fullTableName method to resolv…
…e the table prefix
3611a04
@HavokInspiration HavokInspiration Added a test when a Query is passed as first argument to the TableNam…
…eExpression constructor
c64c0f0
@HavokInspiration HavokInspiration Revert previous commit (3611a04) and implements a more general way of…
… resolving table name to TableNameExpression in Query::join()

While testing, I realized some of the tests did not convert table names to TableNameExpression as a complex array can be given as first argument to Query::join().
This change makes the previous one useless as the present commit is more "general" and takes into account the complex array (with the table name given in the array with the "table" key).
b519e8c
@HavokInspiration
Member

I made an update to the recent addition made in #4017.
While testing I realized some table names in Query::join() where not converted to TableNameExpression as the fullTableName method did not took into account all the forms the first argument of Query::join() could be given.
So I reverted it and implemented a more general way that covers more cases than it did (and hopefully all cases).
I will probably try to refactor this a bit as I'm starting to find the fullTableName method a bit messy...

I also need to update the fullTableName tests. Done.

HavokInspiration added some commits Jul 20, 2014
@HavokInspiration HavokInspiration Update fullTableName method tests
This add a test to take into account the complex array syntax of the first argument that can be given to a Query::join()
f74c667
@HavokInspiration HavokInspiration Merge remote-tracking branch 'upstream/3.0' into 3.0 c267c47
@HavokInspiration HavokInspiration Simplify the purpose of the fullTableName method
While refactoring the fullTableName method, I realized that some Expressions (such as QueryExpression) and even Query object where wrapped in a TableNameExpression which was pointless.
This change simplifies the use of TableNameExpression : it now only wraps strings to prepend the prefix if needed.
9863edd
@HavokInspiration
Member

I pushed more commits where I attempt to simplify the purpose of the fullTableName method and TableNameExpression.
TableNameExpression now only wraps strings as it was pointless for it to wrap QueryExpressions or Query (as when TableNameExpression was rendered as SQL it was only rendering every expression into SQL)

@markstory
Member

I think the fixtures should also apply the connection prefix, which doesn't look like it is hooked up yet. But the changes so far are looking good.

@HavokInspiration
Member

I'm starting to find time to dive in this again.
I'm closing this PR as it's almost 2 months late from the current state of Cake 3.0.
I will start again and open a new one with a fresh fork (I also -very- poorly managed my branches so it will give me the occasion to do things better...)

I'll submit a new PR soon (I hope I can keep that promise).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment