fix foreign query reference, use exists, realias sub-segment query, a…

[galvani]

Fix segment foreign reference bug

Please be sure you are submitting this against the staging branch.

Q	A
Bug fix?	Y
New feature?	N
Automated tests included?	N
Issues addressed (#s or URLs)	none
BC breaks?	N
Deprecations?	N

Description:

Segments with reference to another segment were not built correctly if grouped into condition using in and notIn. This PR fixes it and improves performance by replacing join with exists.

Steps to reproduce the bug and test this PR:

1. Create a segment that defines reference in to multiple other segments that contain different contacts. It should be sufficient that referenced segments contain one contact only.
2. Run and verify that not all contacts that should be there are there.
3. Checkout this PR an rerun
4. Now everything should match.

[Enc3phale]

I have tested this PR with a segment that include segment filter in and notIn but now we have this error on some segments:
[2018-11-19 09:42:31] mautic.NOTICE: Mautic\LeadBundle\Segment\Query\QueryException: The given alias 'l' is not part of any FROM or JOIN clause table. The currently registered aliases are: ASKqQsTW, GUERVocT. (uncaught exception) at /app/bundles/LeadBundle/Segment/Query/QueryException.php line 37 while running console command mautic:segments:update [] []

[galvani]

@Enc3phale could you please paste full backtrace?

[Enc3phale]

@galvani of course:

[Mautic\LeadBundle\Segment\Query\QueryException]                                                                             
  The given alias 'l' is not part of any FROM or JOIN clause table. The currently registered aliases are: gSsyEfzB, ihxXSuVO. 


Exception trace:
 () at /app/bundles/LeadBundle/Segment/Query/QueryException.php:37
 Mautic\LeadBundle\Segment\Query\QueryException::unknownAlias() at /app/bundles/LeadBundle/Segment/Query/QueryBuilder.php:1216
 Mautic\LeadBundle\Segment\Query\QueryBuilder->verifyAllAliasesAreKnown() at /app/bundles/LeadBundle/Segment/Query/QueryBuilder.php:1202
 Mautic\LeadBundle\Segment\Query\QueryBuilder->getFromClauses() at /app/bundles/LeadBundle/Segment/Query/QueryBuilder.php:1160
 Mautic\LeadBundle\Segment\Query\QueryBuilder->getSQLForSelect() at /app/bundles/LeadBundle/Segment/Query/QueryBuilder.php:270
 Mautic\LeadBundle\Segment\Query\QueryBuilder->getSQL() at /app/bundles/LeadBundle/Segment/Query/Filter/SegmentReferenceFilterQueryBuilder.php:124
 Mautic\LeadBundle\Segment\Query\Filter\SegmentReferenceFilterQueryBuilder->applyQuery() at /app/bundles/LeadBundle/Segment/ContactSegmentFilter.php:172
 Mautic\LeadBundle\Segment\ContactSegmentFilter->applyQuery() at /app/bundles/LeadBundle/Segment/Query/ContactSegmentQueryBuilder.php:86
 Mautic\LeadBundle\Segment\Query\ContactSegmentQueryBuilder->assembleContactsSegmentQueryBuilder() at /app/bundles/LeadBundle/Segment/ContactSegmentService.php:222
 Mautic\LeadBundle\Segment\ContactSegmentService->getNewSegmentContactsQuery() at /app/bundles/LeadBundle/Segment/ContactSegmentService.php:74
 Mautic\LeadBundle\Segment\ContactSegmentService->getNewLeadListLeadsCount() at /app/bundles/LeadBundle/Model/ListModel.php:902
 Mautic\LeadBundle\Model\ListModel->rebuildListLeads() at /app/bundles/LeadBundle/Command/UpdateLeadListsCommand.php:86
 Mautic\LeadBundle\Command\UpdateLeadListsCommand->execute() at /vendor/symfony/console/Command/Command.php:241
 Symfony\Component\Console\Command\Command->run() at /vendor/symfony/console/Application.php:861
 Symfony\Component\Console\Application->doRunCommand() at /vendor/symfony/console/Application.php:193
 Symfony\Component\Console\Application->doRun() at /vendor/symfony/framework-bundle/Console/Application.php:84
 Symfony\Bundle\FrameworkBundle\Console\Application->doRun() at /vendor/symfony/console/Application.php:117
 Symfony\Component\Console\Application->run() at mautic_command_CLI.php:62

 mautic:segments:update [-b|--batch-limit [BATCH-LIMIT]] [-m|--max-contacts [MAX-CONTACTS]] [-i|--list-id [LIST-ID]] [-f|--force] [--bypass-locking] [-t|--timeout TIMEOUT] [-x|--lock_mode LOCK_MODE] [-h|--help] [-q|--quiet] [-v|vv|vvv|--verbose] [-V|--version] [--ansi] [--no-ansi] [-n|--no-interaction] [-s|--shell] [--process-isolation] [-e|--env ENV] [--no-debug] [--] <command>

[npracht]

Hi @galvani did you have the opportunity to look at it ?

[npracht]

Hello @galvani any update on this ?

[Enc3phale]

Hi @galvani, in app/bundles/LeadBundle/Segment/Query/Filter/SegmentReferenceFilterQueryBuilder.php

I found this lines

$subSegmentLeadsTableAlias = $this->generateRandomParameterName();
$segmentQueryBuilder->resetQueryParts(['from'])->from(MAUTIC_TABLE_PREFIX.'leads', $subSegmentLeadsTableAlias);

causing the error:
QueryException: The given alias 'l' is not part of any FROM or JOIN clause table.

If I remove lines no more error and segment is compiled, but I don't know if this could be removed.

[npracht]

Hi guys, we tried to work hard on it because we had some issues on segments. The conclusion is that this PR is creating more issues than solving. The root of the issue is coming from somewhere else.

In fact, the issue is: when you delete a segment which is included in another segment (with the filter "segment membership") then the deleted segment keep being saved in the filter of the other segment in the DB ---> that generates tones of issues.
This PR #6566 helps a lot to avoid future issues because you cannot delete a used segment but does not solve issues retroactively (you still need to re-do your old segments).

[galvani]

@npracht we should close it and refuse to merge this PR. I am not longer sure what it was made for. We don't have it in mautic-inc. It's too old.