HPCC-21872 Old index values returned after index is overwritten #12520

richardkchapman · 2019-05-08T10:24:19Z

Clear key cache for any index file that is written by hthor.

For roxie, clear entire index key cache when a file is written (since the id's
stored in the cache do not seem to correspond to physical filenames, it's
non-trivial to clear just the one that is written.

Signed-off-by: Richard Chapman rchapman@hpccsystems.com

Type of change:

This change is a bug fix (non-breaking change which fixes an issue).
This change is a new feature (non-breaking change which adds functionality).
This change improves the code (refactor or other change that does not change the functionality)
This change fixes warnings (the fix does not alter the functionality or the generated code)
This change is a breaking change (fix or feature that will cause existing behavior to change).
This change alters the query API (existing queries will have to be recompiled)

Checklist:

Smoketest:

Send notifications about my Pull Request position in Smoketest queue.
Test my draft Pull Request.

Testing:

using the test file in the Jira, which I added to the test suite.

richardkchapman · 2019-05-08T10:26:11Z

@jakesmith Please review

AttilaVamos · 2019-05-08T11:47:24Z

@richardkchapman there is a regression error (same on all engines) in this PR :

322. Test: indexcachebug.ecl
    --- indexcachebug.xml
    +++ indexcachebug.xml
    @@ -1,3 +1,9 @@
    +<Dataset name='Result 1'>
    +</Dataset>
    +<Dataset name='Result 2'>
    +</Dataset>
    +<Dataset name='Result 3'>
    +</Dataset>
     <Dataset name='Result 4'>
      <Row><fname>Aaron          </fname><lname>Jones          </lname><age>100</age></Row>
      <Row><fname>Adam           </fname><lname>Smith          </lname><age>90</age></Row>
    @@ -9,6 +15,8 @@
      <Row><fname>Edward         </fname><lname>Green          </lname><age>30</age></Row>
      <Row><fname>Egbert         </fname><lname>Sillyname      </lname><age>20</age></Row>
      <Row><fname>Freddy         </fname><lname>Peters         </lname><age>10</age></Row>
    +</Dataset>
    +<Dataset name='Result 5'>
     </Dataset>
     <Dataset name='Result 6'>
      <Row><fname>1              </fname><lname>4280804635     </lname><age>999</age></Row>

Somehow the Smoketest failed to put result report into GitHub. I'm investigating it.

AttilaVamos · 2019-05-08T11:56:55Z

Hmm, by the log:"

{
  "message": "Validation Failed",
  "errors": [
    {
      "resource": "IssueComment",
      "code": "unprocessable",
      "field": "data",
      "message": "User is blocked"
    }
  ],
  "documentation_url": "https://developer.github.com/v3/issues/comments/#create-a-comment"
}

It is strange, based on other test session (PR-12521) after this, finished and reported well.

richardkchapman · 2019-05-08T11:59:12Z

I pushed a fix

AttilaVamos · 2019-05-08T12:01:25Z

I'm watching.

AttilaVamos · 2019-05-08T12:29:33Z

Still strange. Smoketest wasn't able to add test session started comment to this PR.

richardkchapman · 2019-05-08T12:30:57Z

I wonder if it's because I blocked the SmokeTest user (I thought that would stop me getting notifications, but maybe it also stops smoketest from being able to comment on my PRs)

richardkchapman · 2019-05-08T12:33:13Z

I suspect that was it (having read the docs more carefully).

I have unblocked - may also need to re-add as a collaborator. Let me know if you spot anything else I messed up.

AttilaVamos · 2019-05-08T12:33:35Z

Yes, it seems. I tried to add a comment via browser as a Smoketest user and I wasn't able to do that.

HPCCSmoketest · 2019-05-08T12:34:11Z

Test comment

AttilaVamos · 2019-05-08T12:38:26Z

It seems ok now for Smoketest user.
There is still an error (Hthor, Thor):

    322. Test: indexcachebug.ecl
    --- indexcachebug.xml
    +++ indexcachebug.xml
    @@ -1,5 +1,3 @@
    -
    -<Result>
     <Dataset name='Result 1'>
     </Dataset>
     <Dataset name='Result 2'>
    @@ -32,4 +30,3 @@
      <Row><fname>8              </fname><lname>2548245530     </lname><age>992</age></Row>
      <Row><fname>9              </fname><lname>704836619      </lname><age>991</age></Row>
     </Dataset>
    -</Result>

Roxie:

337. Test: indexcachebug.ecl
    --- indexcachebug.xml
    +++ indexcachebug.xml
    @@ -1,5 +1,5 @@
    -
    -<Result>
    +<Warning><Code>4523</Code><Filename>indexcachebug.ecl</Filename><Line>29</Line><Source>eclcc</Source><Message>Neither LIMIT() nor CHOOSEN() supplied for index read on  '~myindex'</Message></Warning>
    +<Warning><Code>4523</Code><Filename>indexcachebug.ecl</Filename><Line>29</Line><Source>eclcc</Source><Message>Neither LIMIT() nor CHOOSEN() supplied for index read on  '~myindex'</Message></Warning>
     <Dataset name='Result 1'>
     </Dataset>
     <Dataset name='Result 2'>
    @@ -32,4 +32,3 @@
      <Row><fname>8              </fname><lname>2548245530     </lname><age>992</age></Row>
      <Row><fname>9              </fname><lname>704836619      </lname><age>991</age></Row>
     </Dataset>
    -</Result>

Clear key cache for any index file that is written by hthor. For roxie, clear entire index key cache when a file is written (since the id's stored in the cache do not seem to correspond to physical filenames, it's non-trivial to clear just the one that is written. Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>

richardkchapman · 2019-05-08T14:37:58Z

I force-pushed another change.

HPCCSmoketest · 2019-05-08T15:38:30Z

Automated Smoketest: ✅
OS: centos 7.4.1708 (Linux 3.10.0-327.28.3.el7.x86_64)
Sha: ca1b812
Build: success
Build: success
Install HPCC Platform
HPCC Start: OK

Unit tests result:

Test	total	passed	elaps
unittest	113	113	36 sec
wutoolTest(Dali)	19	19	2 sec
wutoolTest(Cassandra)	19	19	7 sec

Regression test result:

phase	total	pass	elaps
setup (hthor)	11	11	24 sec (00:00:24)
setup (thor)	11	11	44 sec (00:00:44)
setup (roxie)	11	11	20 sec (00:00:20)
test (hthor)	829	829	164 sec (00:02:44)
test (thor)	754	754	602 sec (00:10:02)
test (roxie)	905	905	218 sec (00:03:38)

HPCC Stop: OK
Time stats:

Prep time	Build time	Package time	Install time	Start time	Test time	Stop time	Summary
29 sec (00:00:29)	211 sec (00:03:31)	0 sec (00:00:00)	3 sec (00:00:03)	18 sec (00:00:18)	1313 sec (00:21:53)	21 sec (00:00:21)	1595 sec (00:26:35)

AttilaVamos · 2019-05-08T15:48:21Z

I don't know where is the Smoketest report comment (by the logs it is created successfully), but this PR is clean now.

richardkchapman · 2019-05-09T07:29:18Z

@jakesmith Please review

jakesmith · 2019-05-10T16:48:29Z

roxie/ccd/ccdserver.cpp

@@ -12290,6 +12290,7 @@ class CRoxieServerIndexWriteActivity : public CRoxieServerInternalSinkActivity,
            duplicateKeyCount = builder->getDuplicateCount();
            cummulativeDuplicateKeyCount += duplicateKeyCount;
            builder->finish(metadata, &fileCrc);
+            clearKeyStoreCache(false);


Curious why this isn't specifically clearing the file just written (as in the hthor change) ?

I address this in the commit comment:

For roxie, clear entire index key cache when a file is written (since the id's
stored in the cache do not seem to correspond to physical filenames, it's
non-trivial to clear just the one that is written.

ah ok, missed that.

jakesmith · 2019-05-10T16:49:09Z

@richardkchapman - one question

richardkchapman · 2019-05-13T11:05:50Z

@jakesmith See answer

jakesmith · 2019-05-13T11:37:02Z

@richardkchapman - good to merge.

richardkchapman · 2019-05-13T15:07:36Z

@ghalliday Please merge

richardkchapman force-pushed the hpcc21872 branch from 3141be8 to 8356f34 Compare May 8, 2019 10:25

richardkchapman force-pushed the hpcc21872 branch from 8356f34 to cc02486 Compare May 8, 2019 11:58

richardkchapman force-pushed the hpcc21872 branch from cc02486 to ca1b812 Compare May 8, 2019 14:36

jakesmith reviewed May 10, 2019

View reviewed changes

ghalliday merged commit 18a7e05 into hpcc-systems:candidate-7.2.x May 14, 2019

richardkchapman deleted the hpcc21872 branch February 15, 2020 11:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HPCC-21872 Old index values returned after index is overwritten #12520

HPCC-21872 Old index values returned after index is overwritten #12520

richardkchapman commented May 8, 2019

richardkchapman commented May 8, 2019

AttilaVamos commented May 8, 2019

AttilaVamos commented May 8, 2019

richardkchapman commented May 8, 2019 •

edited

AttilaVamos commented May 8, 2019

AttilaVamos commented May 8, 2019

richardkchapman commented May 8, 2019

richardkchapman commented May 8, 2019

AttilaVamos commented May 8, 2019

HPCCSmoketest commented May 8, 2019

AttilaVamos commented May 8, 2019

richardkchapman commented May 8, 2019

HPCCSmoketest commented May 8, 2019

AttilaVamos commented May 8, 2019

richardkchapman commented May 9, 2019

jakesmith May 10, 2019

richardkchapman May 13, 2019

jakesmith May 13, 2019

jakesmith commented May 10, 2019

richardkchapman commented May 13, 2019

jakesmith commented May 13, 2019

richardkchapman commented May 13, 2019

HPCC-21872 Old index values returned after index is overwritten #12520

HPCC-21872 Old index values returned after index is overwritten #12520

Conversation

richardkchapman commented May 8, 2019

Type of change:

Checklist:

Smoketest:

Testing:

richardkchapman commented May 8, 2019

AttilaVamos commented May 8, 2019

AttilaVamos commented May 8, 2019

richardkchapman commented May 8, 2019 • edited

AttilaVamos commented May 8, 2019

AttilaVamos commented May 8, 2019

richardkchapman commented May 8, 2019

richardkchapman commented May 8, 2019

AttilaVamos commented May 8, 2019

HPCCSmoketest commented May 8, 2019

AttilaVamos commented May 8, 2019

richardkchapman commented May 8, 2019

HPCCSmoketest commented May 8, 2019

AttilaVamos commented May 8, 2019

richardkchapman commented May 9, 2019

jakesmith May 10, 2019

Choose a reason for hiding this comment

richardkchapman May 13, 2019

Choose a reason for hiding this comment

jakesmith May 13, 2019

Choose a reason for hiding this comment

jakesmith commented May 10, 2019

richardkchapman commented May 13, 2019

jakesmith commented May 13, 2019

richardkchapman commented May 13, 2019

richardkchapman commented May 8, 2019 •

edited