Add format option for add_locale processor #4106

martinscholz83 · 2017-04-25T10:30:19Z

Like discussed in #3974, this PR add the option to format the timezoneoutput. I have decided to just use abbrevation and offset as options. Because is not so trivial to format time to a canonical_id. Here is a discussion about the topic. One thing which have to decide is should be format option required or should be abbrevation the default option.

elasticmachine · 2017-04-25T10:32:02Z

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run.

elasticmachine · 2017-04-25T10:32:07Z

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run.

andrewkroh

Looks like you are off to a good start. Thanks for working on this!

andrewkroh · 2017-04-26T20:23:01Z

libbeat/processors/add_locale/add_locale.go

+}
+
+func (t TimezoneFormat) String() string {
+	return timezoneFormats[t]


It seems like this could easily result in a panic with something like TimezoneFormat(-1).String(). I think a more common approach is to store a map[TimezoneFormat]string. But you can use an array like you did, but you just need to bounds checking on t like (t > 0 && t < len(timezoneFormats)).

andrewkroh · 2017-04-26T20:25:34Z

libbeat/processors/add_locale/add_locale.go

-	return addLocale{}, nil
+	config := struct {
+		Format string `config:"format" validate:"required"`
+	}{}


Looks like this is missing the default value initialization.

That's what i asked. How do you think should that processor work

-add_locale:

with default abbrevation
or

-add_locale: format: "abbrevation/offset"

make format as required.

I think it should default to offset. IMO that would be easier to use and not require someone to have a lookup table of names to numbers and would account for DST changes.

andrewkroh · 2017-04-26T20:25:52Z

libbeat/processors/add_locale/add_locale.go

+	}{}
+	err := c.Unpack(&config)
+	if err != nil {
+		return nil, fmt.Errorf("fail to unpack the include_fields configuration: %s", err)


s/include_fields/add_locale/

andrewkroh · 2017-04-26T20:28:56Z

libbeat/processors/add_locale/add_locale.go

+	case "offset":
+		loc.TimezoneFormat = Offset
+	default:
+		return nil, errors.New("given format is not valid")


More details in the error would be better for users running -configcheck. Things like: What is the invalid value that was given? What is the thing that caused the error (add_locale)? How should the error be fixed (provide the two allowed values)?

andrewkroh · 2017-04-26T20:30:42Z

libbeat/processors/add_locale/add_locale.go

 }

 func (l addLocale) Run(event common.MapStr) (common.MapStr, error) {
-	zone, _ := time.Now().Zone()
-	event.Put("beat.timezone", zone)
+	format := l.Format()


How do the old benchmarks compare to the new benchmarks now that Format() is called for each event? It would be good to test with both Abbreviation and Offset.

andrewkroh · 2017-04-26T20:33:16Z

libbeat/processors/add_locale/add_locale.go

+	case Abbrevation:
+		fmt, _ = tm.Zone()
+	case Offset:
+		//loc, _ := time.LoadLocation("America/Belize")


This should be removed.

andrewkroh · 2017-04-26T21:18:48Z

libbeat/processors/add_locale/add_locale.go

+		}
+
+		fmt += strconv.Itoa(offset)
+		fmt += ":00"


Timezone offsets do not all fall on one hour boundaries. See https://en.wikipedia.org/wiki/List_of_tz_database_time_zones

Here's an example calculation: https://play.golang.org/p/iuiuSxMZ1Q

This will need some test cases.

What is the correct form for positive and negative values to display.
11:45, 02:45 and -11:45, -01:45 or 11:45, 2:45 and -11:45, -1:45. Its about the leading zero and the positive and negative sign.

From the wiki page I linked above:

The UTC offsets (columns 5 and 6) are parsed from the "continent files" and accordingly ISO 6709 are in the format {+|-}hh:mm

So I'd say go with {+|-}hh:mm.

andrewkroh · 2017-04-26T21:21:18Z

jenkins, test it

ruflin · 2017-05-04T12:31:06Z

@maddin2016 One of the tests is failing: add_locale_test.go:38: fail to unpack the include_fields configuration: missing required field accessing 'format'

martinscholz83 · 2017-05-04T12:38:30Z

Hi @ruflin, that's right. There 2 questions which are unanswered. Then i can continue.

How do you think this processor should work

-add_locale:

with default abbrevation
or

-add_locale:
  format: "abbrevation/offset"

make format as required.

What is the correct form for positive and negative values to display.
11:45, 02:45 and -11:45, -01:45 or 11:45, 2:45 and -11:45, -1:45. Its about the leading zero and the positive and negative sign.

ruflin · 2017-05-09T06:40:38Z

@monicasarbu @andrewkroh Can one of you comment on the above?

martinscholz83 · 2017-05-09T07:17:49Z

Hi @ruflin. @andrewkroh has answered my question. Can you start the integration tests again. Something went wrong with travis. Thanks.

ruflin · 2017-05-09T07:25:00Z

Nice, just restarted it.

ruflin · 2017-05-09T07:25:05Z

jenkins, test it

elasticmachine · 2017-05-09T07:55:02Z

libbeat/processors/add_locale/add_locale.go

+	TimezoneFormat TimezoneFormat
+}
+
+type TimezoneFormat int


[golint] _{reported by reviewdog 🐶}
exported type TimezoneFormat should have comment or be unexported

elasticmachine · 2017-05-09T07:55:02Z

libbeat/processors/add_locale/add_locale.go

+type TimezoneFormat int
+
+const (
+	Abbrevation TimezoneFormat = iota


[golint] _{reported by reviewdog 🐶}
exported const Abbrevation should have comment (or a comment on this block) or be unexported

dedemorton · 2017-05-16T20:53:22Z

@andrewkroh Shouldn't there be a doc change for this, too?

andrewkroh · 2017-05-16T20:54:35Z

I updated the docs in #4330

Add format option for timezone

8152964

ruflin added :Processors review labels Apr 25, 2017

andrewkroh reviewed Apr 26, 2017

View reviewed changes

martinscholz83 added 2 commits April 27, 2017 10:18

Change String() method for TimezoneFormat

3cb6c44

Change error message

3047b5a

martinscholz83 added 3 commits May 5, 2017 15:00

Make offset default format option

9b95e7b

Improve Format() function

98fb46e

Fix test

f607c8d

elasticmachine reviewed May 9, 2017

View reviewed changes

martinscholz83 added 2 commits May 9, 2017 11:41

Add descriptions

c294778

Change Format() overload

a5d04e2

andrewkroh approved these changes May 9, 2017

View reviewed changes

andrewkroh merged commit 79ebc5c into elastic:master May 9, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add format option for add_locale processor #4106

Add format option for add_locale processor #4106

martinscholz83 commented Apr 25, 2017

elasticmachine commented Apr 25, 2017

elasticmachine commented Apr 25, 2017

andrewkroh left a comment

andrewkroh Apr 26, 2017

andrewkroh Apr 26, 2017

martinscholz83 Apr 27, 2017 •

edited

Loading

andrewkroh May 4, 2017 •

edited

Loading

andrewkroh Apr 26, 2017

andrewkroh Apr 26, 2017

andrewkroh Apr 26, 2017

andrewkroh Apr 26, 2017

martinscholz83 Apr 27, 2017

andrewkroh Apr 26, 2017

martinscholz83 Apr 27, 2017

andrewkroh May 4, 2017

andrewkroh commented Apr 26, 2017

ruflin commented May 4, 2017

martinscholz83 commented May 4, 2017

ruflin commented May 9, 2017

martinscholz83 commented May 9, 2017

ruflin commented May 9, 2017

ruflin commented May 9, 2017

elasticmachine May 9, 2017

elasticmachine May 9, 2017

dedemorton commented May 16, 2017

andrewkroh commented May 16, 2017

Add format option for add_locale processor #4106

Add format option for add_locale processor #4106

Conversation

martinscholz83 commented Apr 25, 2017

elasticmachine commented Apr 25, 2017

elasticmachine commented Apr 25, 2017

andrewkroh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

martinscholz83 Apr 27, 2017 • edited Loading

Choose a reason for hiding this comment

andrewkroh May 4, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewkroh commented Apr 26, 2017

ruflin commented May 4, 2017

martinscholz83 commented May 4, 2017

ruflin commented May 9, 2017

martinscholz83 commented May 9, 2017

ruflin commented May 9, 2017

ruflin commented May 9, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dedemorton commented May 16, 2017

andrewkroh commented May 16, 2017

martinscholz83 Apr 27, 2017 •

edited

Loading

andrewkroh May 4, 2017 •

edited

Loading