New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: Add filter to lift keys from a map or nest keys in a map #531

Merged
merged 2 commits into from May 17, 2018

Conversation

Projects
None yet
2 participants
@michiel
Contributor

michiel commented Mar 8, 2018

This adds a new filter, called 'nest', that allows keys to be,

  • Nested under another key using a wildcard
  • Lifted up from a nest

Examples,

Nest 1

[INPUT]
    Name mem
    Tag  mem.local

[OUTPUT]
    Name  stdout
    Match *

[FILTER]
    Name nest
    Match *
    Operation nest
    Wildcard Mem.*
    Nest_under LAYER1

Output,

[0] mem.local: [1524795567.010244325, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "LAYER1"=>{"Mem.total"=>4050908, "Mem.used"=>1101372, "Mem.free"=>2949536}}]

{
  "Swap.total"=>1046524,
  "Swap.used"=>0,
  "Swap.free"=>1046524,
  "LAYER1"=>{
   "Mem.total"=>4050908,
   "Mem.used"=>1101372,
   "Mem.free"=>2949536
  }
}

Nest 1 - Lift 1

[INPUT]
    Name mem
    Tag  mem.local

[OUTPUT]
    Name  stdout
    Match *

[FILTER]
    Name nest
    Match *
    Operation nest
    Wildcard Mem.*
    Nest_under LAYER1

[FILTER]
    Name nest
    Match *
    Operation lift
    Nested_under LAYER1

Output,

[0] mem.local: [1524795757.002111309, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "Mem.total"=>4050908, "Mem.used"=>1112044, "Mem.free"=>2938864}]

{
  "Swap.total"=>1046524, 
  "Swap.used"=>0,
  "Swap.free"=>1046524, 
  "Mem.total"=>4050908, 
  "Mem.used"=>1112044, 
  "Mem.free"=>2938864
}

Nest 1/2/3

[INPUT]
    Name mem
    Tag  mem.local

[OUTPUT]
    Name  stdout
    Match *

[FILTER]
    Name nest
    Match *
    Operation nest
    Wildcard Mem.*
    Nest_under LAYER1

[FILTER]
    Name nest
    Match *
    Operation nest
    Wildcard LAYER1*
    Nest_under LAYER2

[FILTER]
    Name nest
    Match *
    Operation nest
    Wildcard LAYER2*
    Nest_under LAYER3

Output,

 [0] mem.local: [1524795923.009867831, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "LAYER3"=>{"LAYER2"=>{"LAYER1"=>{"Mem.total"=>4050908, "Mem.used"=>1112036, "Mem.free"=>2938872}}}}]

 {
   "Swap.total"=>1046524,
   "Swap.used"=>0,
   "Swap.free"=>1046524,
   "LAYER3"=>{
     "LAYER2"=>{
       "LAYER1"=>{
         "Mem.total"=>4050908,
         "Mem.used"=>1112036,
         "Mem.free"=>2938872
       }
     }
   }
 }

Nest 1/2/3 - Lift 3/2

[INPUT]
    Name mem
    Tag  mem.local

[OUTPUT]
    Name  stdout
    Match *

[FILTER]
    Name nest
    Match *
    Operation nest
    Wildcard Mem.*
    Nest_under LAYER1

[FILTER]
    Name nest
    Match *
    Operation nest
    Wildcard LAYER1*
    Nest_under LAYER2

[FILTER]
    Name nest
    Match *
    Operation nest
    Wildcard LAYER2*
    Nest_under LAYER3

[FILTER]
    Name nest
    Match *
    Operation lift
    Nested_under LAYER3

[FILTER]
    Name nest
    Match *
    Operation lift
    Nested_under LAYER2

Output,

[0] mem.local: [1524795834.019236872, {"Swap.total"=>1046524, "Swap.used"=>0, "Swap.free"=>1046524, "LAYER1"=>{"Mem.total"=>4050908, "Mem.used"=>1112152, "Mem.free"=>2938756}}]

{
  "Swap.total"=>1046524,
  "Swap.used"=>0,
  "Swap.free"=>1046524, 
  "LAYER1"=>{
    "Mem.total"=>4050908, 
    "Mem.used"=>1112152, 
    "Mem.free"=>2938756
  }
}
@@ -0,0 +1,284 @@
/* -*- Mode: C; tab-width: 4; indent-tabs-mode: nil; c-basic-offset: 4 -*- */
/*

This comment has been minimized.

@edsiper

edsiper Apr 2, 2018

Contributor

Please add missing header here:

/*  Fluent Bit
 *  ==========
 *  Copyright (C) 2015-2017 Treasure Data Inc.

This comment has been minimized.

@michiel

michiel Apr 4, 2018

Contributor

Added in 77ac171

@michiel michiel force-pushed the michiel:feature/add_nest_filter branch from 030fadd to 94780d5 Apr 9, 2018

@michiel

This comment has been minimized.

Contributor

michiel commented Apr 9, 2018

Documentation PR added at fluent/fluent-bit-docs#62

@michiel michiel force-pushed the michiel:feature/add_nest_filter branch from 94780d5 to eedb6f1 Apr 26, 2018

@michiel michiel changed the title from feature: Add filter to nest specific keys under another key to feature: Add filter to lift keys from a map or nest keys in a map May 2, 2018

@michiel michiel force-pushed the michiel:feature/add_nest_filter branch from 00cd563 to 58e6860 May 2, 2018

@michiel

This comment has been minimized.

Contributor

michiel commented May 3, 2018

Documentation PR updated to reflect addition of 'lift' operation

See https://github.com/michiel/fluent-bit-docs/blob/feature/add_filter_nest/filter/nest.md

filter: Add new nest filter
This adds a filter, called 'nest'. It allows you to,

 - Nest a set of key-values pairs under a new key
 - Lift a set of key-value pairs up to the top level

Signed-off-by: Michiel Kalkman <mkalkman@shelde.com>

@michiel michiel force-pushed the michiel:feature/add_nest_filter branch from f14545d to 89183b8 May 9, 2018

@edsiper edsiper merged commit e97a293 into fluent:master May 17, 2018

1 of 2 checks passed

continuous-integration/travis-ci/pr The Travis CI build is in progress
Details
DCO All commits have a DCO sign-off from the author
Details
@edsiper

This comment has been minimized.

Contributor

edsiper commented May 17, 2018

thanks!

edsiper added a commit that referenced this pull request May 21, 2018

filter_nest: new filter to handle nest k/v (#531)
This adds a filter, called 'nest'. It allows you to,

 - Nest a set of key-values pairs under a new key
 - Lift a set of key-value pairs up to the top level

Signed-off-by: Michiel Kalkman <mkalkman@shelde.com>
Signed-off-by: Eduardo Silva <eduardo@treasure-data.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment