Feat/plugins schemas to new db #3778

kikito · 2018-09-16T18:05:32Z

No description provided.

hishamhm · 2018-10-02T12:48:49Z

Reviewing commit by commit!

Leaving a note here since there is no "general comment on commit": please rename fix(schema) default values for structs to fix(schema) default values for records to avoid confusion.

hishamhm · 2018-10-02T12:50:46Z

kong/db/schema/init.lua

@@ -270,6 +270,9 @@ Schema.validators = {
  end,

  uuid = function(value)
+    if value == "" then
+      return true
+    end


This does look odd, and the commit says "Check if this is needed"... which plugin test needs this to pass?

Looks like this is related to config.anonymous. I see you added is_present in the plugins to support both "" and ngx.null, I guess to avoid a migration (always a good goal!).

I wonder if we could get rid of these empty strings via process_auto_fields instead. Perhaps with a legacy annotation?

-- in process_auto_fields (for field.auto == false case only) if field.legacy and field.uuid and output[key] == "" then output[key] == ngx.null end

This would always clean up the data once it comes from the DB (it's a bit of a "migration transformation"). The plugins wouldn't need the is_present tests and this change to the uuid validator wouldn't be necessary (ie, we'd avoid messing up all UUID fields because of anonymous). What do you think?

I like that! It didn't occur to me to do it that way.

kong/db/schema/init.lua

hishamhm · 2018-10-02T13:00:20Z

kong/db/schema/init.lua

        output[key] = utils.random_string()
      end
    end

    if context == "update" then
      output[key] = adjust_field_for_update(field, output[key], nulls)
+


minor: the style guide says this is not necessary for one-liner if/then/else blocks

ok removed that one

kong/db/schema/init.lua

hishamhm · 2018-10-02T13:12:40Z

kong/plugins/bot-detection/schema.lua

+        fields = {
+          { whitelist = {
+              type = "array",
+              elements = { type = "string", is_regex = true, match = ".*" },


I understand this match rule was already present in the old schema, but it does nothing useful ("match zero or more of any character"), right?

kong/plugins/datadog/schema.lua

kong/plugins/request-termination/schema.lua

hishamhm · 2018-10-02T14:17:37Z

The "pr" build of Travis seems to be failing over and over in the C*2 integration job with a weird error in the datadog test suite. Currently the are conflicts with next, and @bungle has fixed a flakiness problem in the datadog tests. Perhaps this will be fixed after this PR merges cleanly again?

hishamhm

All set, but waiting for master to be merged into next so we can remove the tests(helpers) workaround on this

Entities already had non-nullable records by default since 30789e7. However, plugin subschemas did not go through the same preprocessing. This adds a new function `Entity.new_subschema` that performs that same preprocessing.

We cannot use `between' because sometimes the condition is non-inclusive (`gt = 0` does not include 0)

This calls `handle_missing_value` when propagating defaults for records so that non-nullable records (as in Entities) get their sub-tables created. Also refactors both adjustment functions into one (the original version of this patch missed the other function, so this avoids it happening again). Co-authored-by: Enrique García Cota <kikito@gmail.com>

UUIDs created with the old schema stored an empty string to mean "value not present". We now use an explicit `null` for that. This `legacy` annotation allows the schema library to transform existing rows stored in the old format to use the new format transparently. Co-authored-by: Hisham Muhammad <hisham@gobolinux.org>

hishamhm · 2018-10-04T19:36:04Z

Merged with a merge commit, as we do for major changes

kikito added the pr/do not merge label Sep 16, 2018

kikito force-pushed the feat/plugins-schemas-to-new-db branch 10 times, most recently from 64bb88b to 6675897 Compare September 20, 2018 23:27

This was referenced Sep 21, 2018

fix(request-termination) allow null value for body or message #3784

Closed

FEATURE - Rate limit by key #3779

Closed

kikito force-pushed the feat/plugins-schemas-to-new-db branch 6 times, most recently from aedbf57 to aa8742b Compare September 21, 2018 14:03

kikito removed the pr/do not merge label Sep 21, 2018

kikito changed the title ~~WIP - Feat/plugins schemas to new db~~ Feat/plugins schemas to new db Sep 21, 2018

kikito force-pushed the feat/plugins-schemas-to-new-db branch 2 times, most recently from 13954de to 6db701c Compare September 21, 2018 14:57

thibaultcha added the pr/please review label Sep 21, 2018

hishamhm reviewed Oct 2, 2018

View reviewed changes

kikito force-pushed the feat/plugins-schemas-to-new-db branch 3 times, most recently from d0f00b4 to a6880cd Compare October 3, 2018 17:22

kikito force-pushed the feat/plugins-schemas-to-new-db branch from a6880cd to 4cb6341 Compare October 3, 2018 17:25

hishamhm force-pushed the feat/plugins-schemas-to-new-db branch from 4cb6341 to 131a8f1 Compare October 3, 2018 17:58

kikito force-pushed the feat/plugins-schemas-to-new-db branch from 131a8f1 to 23169b5 Compare October 3, 2018 19:24

hishamhm force-pushed the feat/plugins-schemas-to-new-db branch from 23169b5 to 9f7bc77 Compare October 3, 2018 19:26

kikito force-pushed the feat/plugins-schemas-to-new-db branch from 9f7bc77 to df89f46 Compare October 4, 2018 11:59

hishamhm force-pushed the feat/plugins-schemas-to-new-db branch from 4fac07c to eb91de2 Compare October 4, 2018 17:44

hishamhm added pr/ready (but hold merge) No more concerns, but do not merge yet (probably a conflict of interest with another PR or release) and removed pr/please review labels Oct 4, 2018

hishamhm approved these changes Oct 4, 2018

View reviewed changes

hishamhm and others added 8 commits October 4, 2018 15:52

fix(schema) make records non-nullable in plugin subschemas

d72dd94

Entities already had non-nullable records by default since 30789e7. However, plugin subschemas did not go through the same preprocessing. This adds a new function `Entity.new_subschema` that performs that same preprocessing.

feat(schema) add gt validator

35f2ed6

We cannot use `between' because sometimes the condition is non-inclusive (`gt = 0` does not include 0)

feat(schema) add contains validator

a18b193

fix(schema) VALIDATION error was not reachable

df8e59d

feat(schemas) move schemas of plugins to new DAO format

b415ea7

tests(plugins) fix tests after moving schemas to the new DAO

9e0d1c4

hishamhm force-pushed the feat/plugins-schemas-to-new-db branch from eb91de2 to 9e0d1c4 Compare October 4, 2018 19:02

hishamhm merged commit 9cda865 into next Oct 4, 2018

hishamhm deleted the feat/plugins-schemas-to-new-db branch October 4, 2018 19:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat/plugins schemas to new db #3778

Feat/plugins schemas to new db #3778

kikito commented Sep 16, 2018

hishamhm commented Oct 2, 2018

hishamhm Oct 2, 2018

hishamhm Oct 2, 2018

kikito Oct 3, 2018

hishamhm Oct 2, 2018

kikito Oct 3, 2018

hishamhm Oct 2, 2018

hishamhm commented Oct 2, 2018 •

edited

Loading

hishamhm left a comment

hishamhm commented Oct 4, 2018

Feat/plugins schemas to new db #3778

Feat/plugins schemas to new db #3778

Conversation

kikito commented Sep 16, 2018

hishamhm commented Oct 2, 2018

hishamhm Oct 2, 2018

Choose a reason for hiding this comment

hishamhm Oct 2, 2018

Choose a reason for hiding this comment

kikito Oct 3, 2018

Choose a reason for hiding this comment

hishamhm Oct 2, 2018

Choose a reason for hiding this comment

kikito Oct 3, 2018

Choose a reason for hiding this comment

hishamhm Oct 2, 2018

Choose a reason for hiding this comment

hishamhm commented Oct 2, 2018 • edited Loading

hishamhm left a comment

Choose a reason for hiding this comment

hishamhm commented Oct 4, 2018

hishamhm commented Oct 2, 2018 •

edited

Loading