fix: allow disabling field metadata cache

[znep]

Clients accessing very dynamic schemas can have issues with the
field metadata cache getting stale. This change allows configuring the
databaseMetadataCacheFields property to 0 to disable the cache and
avoid these issues where necessary. This behaviour was already
documented, however did not actually work as the codepath assumed
it could retrieve the fields from the cache.

[davecramer]

How do we test this ?

[znep]

@davecramer Good question, I was unsure the extent of testing that is typical for modifying previously untested code paths in this project so was going to add a follow-up comment asking for input.

I can certainly make a test that creates a table, reads from it, modifies the data type of a column, then reads from it again and ensures that the correct new type is returned when caching is disabled and the old "incorrect" value is returned when caching is enabled.

Does that seem sufficient? It wouldn't be full coverage of the caching codepath but we would have a test that fails before the fix and passes after, and one that "passes" both before and after the fix.

[davecramer]

@znep we'd certainly like to make sure that the codepath has some testing and that this couldn't occur again due to some change in the future.

Your proposed test sounds like a great start!

[vlsi]

As far as I read the code, a mere databaseMetadataCacheFields=0 + regular metadata test should be enough.

That is it makes sense to add @Parameterized to some of the existing tests and parameterize over databaseMetadataCacheFields values like we do for binary, etc, etc.

@znep , any luck with pushing this forward?

[davecramer]

@znep ping ?

[znep]

Hi guys,

I haven't had a chance to circle back to this yet but still plan to this week, I'll take a look at @vlsi 's suggestion.

[codecov-io]

Codecov Report

Merging #1052 into master will increase coverage by 0.04%.
The diff coverage is 100%.

@@             Coverage Diff              @@
##             master    #1052      +/-   ##
============================================
+ Coverage     69.14%   69.18%   +0.04%     
- Complexity     3790     3797       +7     
============================================
  Files           170      171       +1     
  Lines         15760    15765       +5     
  Branches       2578     2578              
============================================
+ Hits          10897    10907      +10     
+ Misses         3621     3618       -3     
+ Partials       1242     1240       -2

[znep]

@davecramer @vlsi None of the existing ResultSetMetaDataTest tests seem to have quite the right cases to repro this, so I added a new test case and parameterized the test suite.

I verified the tests fail as I expect without my fix.

[znep]

@davecramer @vlsi ping, anything else I should add here? Thanks.

[vlsi]

@znep , it looks good, however I wonder if fieldInfoFetched should still be set to true after metadata fetch from the database.

I think the end of populateFieldsWithMetadata should read fieldInfoFetched |= allOk;

What do you think?

[znep]

@vlsi Good catch, thanks for looking. I agree it should be, this isn't a regression from my changes and is a pretty minor optimization (it doesn't actually refetch it from the cache or db, just iterates through all the fields and sees they are already set) but I pushed a fix.

I'm not seeing a way to add test coverage for this without some fairly major structural changes.

(build currently failing due to travis apt-get issues...)

[vlsi]

As a follow-up it might make sense to add metadata cache invalidation on ALTER statements.