HPCC-22869 Lightweight join must always ensure sides are ungrouped

[jakesmith]

Signed-off-by: Jake Smith jake.smith@lexisnexisrisk.com

Type of change:

• This change is a bug fix (non-breaking change which fixes an issue).
• This change is a new feature (non-breaking change which adds functionality).
• This change improves the code (refactor or other change that does not change the functionality)
• This change fixes warnings (the fix does not alter the functionality or the generated code)
• This change is a breaking change (fix or feature that will cause existing behavior to change).
• This change alters the query API (existing queries will have to be recompiled)

Checklist:

• My code follows the code style of this project.
  • My code does not create any new warnings from compiler, build system, or lint.
• The commit message is properly formatted and free of typos.
  • The commit message title makes sense in a changelog, by itself.
  • The commit is signed.
• My change requires a change to the documentation.
  • I have updated the documentation accordingly, or...
  • I have created a JIRA ticket to update the documentation.
  • Any new interfaces or exported functions are appropriately commented.
• I have read the CONTRIBUTORS document.
• The change has been fully tested:
  • I have added tests to cover my changes.
  • All new and existing tests passed.
  • I have checked that this change does not introduce memory leaks.
  • I have used Valgrind or similar tools to check for potential issues.
• I have given due consideration to all of the following potential concerns:
  • Scalability
  • Performance
  • Security
  • Thread-safety
  • Premature optimization
  • Existing deployed queries will not be broken
  • This change fixes the problem, not just the symptom
  • The target branch of this pull request is appropriate for such a change.
• There are no similar instances of the same problem that should be addressed
  • I have addressed them here
  • I have raised JIRA issues to address them separately
• This is a user interface / front-end modification
  • I have tested my changes in multiple modern browsers
  • The component(s) render as expected

Smoketest:

• Send notifications about my Pull Request position in Smoketest queue.
• Test my draft Pull Request.

Testing:

[hpcc-jirabot]

https://track.hpccsystems.com/browse/HPCC-22869
Jira updated

[jakesmith]

@ghalliday - pushed to add a regression test.
Please review

[jakesmith]

@ghalliday @richardkchapman - should be this be 7.2.x?

[richardkchapman]

Has any 7.2.x user reported it?

[jakesmith]

yes, the only report so far was on 7.2.36

[richardkchapman]

Bit nervous about adding bugfixes that change the result in a point release. Is there a workaround?

[jakesmith]

They succeeded in working around the bug by adding a DEGROUP on the RHS of the join - not sure if guaranteed to remain once optimized though.

[ghalliday]

@jakesmith a couple of questions

[ghalliday]

Is there any way of only doing this if the stream is grouped?

[jakesmith]

I guess it is an extra virtual call per row,
and I think I can test input->isGrouped() to make it conditional

[ghalliday]

what happens if it is run in roxie/hthor?

[jakesmith]

it will run and match, but appears not generate a lightweight join..

[jakesmith]

@ghalliday - have changed it so conditional on isGrouped()

Not sure whether the new regression query should bother to run on hthor or roxie ?

[ghalliday]

The change looks good to me. I probably would enable the tests for roxie/hthor, even though they don't generate that specific activity kind.
Need to decide which version to merge this in to.

[jakesmith]

@ghalliday - have changed the regression test so also runs on hthor and roxie (and squashed that change in).

[HPCCSmoketest]

Automated Smoketest: ✅
OS: centos 7.6.1810 (Linux 3.10.0-327.28.3.el7.x86_64)
GCC:gcc (GCC) 7.3.1 20180303 (Red Hat 7.3.1-5)
Sha: f5e42b6
Build: success
Build: success
Install HPCC Platform
HPCC Start: OK

Unit tests result:

Test	total	passed	failed	errors	timeout	elaps
unittest	113	113	0	0	0	35 sec
wutoolTest(Dali)	19	19	0	0	0	3 sec
wutoolTest(Cassandra)	19	19	0	0	0	7 sec

Regression test result:

phase	total	pass	fail	elaps
setup (hthor)	11	11	0	23 sec (00:00:23)
setup (thor)	11	11	0	45 sec (00:00:45)
setup (roxie)	11	11	0	24 sec (00:00:24)
test (hthor)	852	852	0	174 sec (00:02:54)
test (thor)	775	775	0	700 sec (00:11:40)
test (roxie)	928	928	0	235 sec (00:03:55)

HPCC Stop: OK
Time stats:

Prep time	Build time	Package time	Install time	Start time	Test time	Stop time	Summary
35 sec (00:00:35)	252 sec (00:04:12)	0 sec (00:00:00)	3 sec (00:00:03)	19 sec (00:00:19)	1423 sec (00:23:43)	20 sec (00:00:20)	1752 sec (00:29:12)