[file-sd-part-1] Added file sd to query #546

ivan-valkov · 2018-10-01T10:20:42Z

Changes

Added file sd to query. You can specify files to watch via a command line flag. Then the querier will load stores' addresses from the files whenever the files are updated.

We are reusing Prometheus' file sd.

Verification

Tested in a following PR that adds the different SD options to the tests - (#554)

bwplotka

Nice one, thanks! So EXCITED

bwplotka · 2018-10-05T14:04:47Z

cmd/thanos/query.go

@@ -64,6 +66,9 @@ func registerQuery(m map[string]setupFunc, app *kingpin.Application, name string
 	stores := cmd.Flag("store", "Addresses of statically configured store API servers (repeatable).").
 		PlaceHolder("<store>").Strings()

+	filesToWatch := cmd.Flag("filesd", "Path to file that contain addresses of store API servers (repeatable).").


Note something about globbing option.

store-sd-files

bwplotka · 2018-10-05T14:08:01Z

cmd/thanos/query.go

@@ -241,6 +258,15 @@ func runQuery(

 					specs = append(specs, &gossipSpec{id: id, addr: ps.StoreAPIAddr, peer: peer})
 				}
+
+				addrFromFileSD.mtx.Lock()


why not doing just addrFromFileSD.Lock()

bwplotka · 2018-10-05T14:09:22Z

cmd/thanos/query.go

@@ -264,6 +290,48 @@ func runQuery(
 			stores.Close()
 		})
 	}
+	// Run File Service Discovery and update the store set when the files are modified


missed trailing period

bwplotka · 2018-10-05T14:09:48Z

cmd/thanos/query.go

@@ -264,6 +290,48 @@ func runQuery(
 			stores.Close()
 		})
 	}
+	// Run File Service Discovery and update the store set when the files are modified
+	{


if fileSD != nil {

bwplotka · 2018-10-05T14:13:51Z

cmd/thanos/query.go

+	{
+		if fileSD != nil {
+			var fileSDUpdates chan *discovery.Discoverable
+			ctx, cancel := context.WithCancel(context.Background())


never share contexts between two run groups

bwplotka · 2018-10-05T14:14:14Z

cmd/thanos/query.go

+							continue
+						}
+						// TODO(ivan): resolve dns here maybe?
+						addrFromFileSD.update(update.Source, update.Services)


`s/addrFromFileSD/cache

bwplotka · 2018-10-05T14:15:22Z

cmd/thanos/query.go

+				}
+			}, func(error) {
+				cancel()
+				stores.Close()


Maybe it was closed before?

bwplotka · 2018-10-05T14:16:28Z

cmd/thanos/query.go

+							continue
+						}
+						// TODO(ivan): resolve dns here maybe?
+						addrFromFileSD.update(update.Source, update.Services)


validate for dups?

domgreen

Looks good 👍 like the reuse of Prom code :)

domgreen · 2018-10-11T11:16:34Z

cmd/thanos/query.go

@@ -271,6 +302,46 @@ func runQuery(
 			stores.Close()
 		})
 	}
+	// Run File Service Discovery and update the store set when the files are modified.
+	if fileSD != nil {


minor nit - did we want to put this in a block so we can keep ctx, cancel as we have done in other blocks?

It is in block, thanks for if (: right?

Yeah its fine, i didnt think about the indenting

domgreen · 2018-10-11T11:25:31Z

docs/components/query.md

@@ -129,6 +129,10 @@ Flags:
                                 info endpoint (repeated).
      --store=<store> ...        Addresses of statically configured store API
                                 servers (repeatable).
+      --store-sd-file=<path> ...  


nit - would change this to match prom file_sd_config https://prometheus.io/docs/prometheus/latest/configuration/configuration/#file_sd_config so maybe store.file-sd-config?

could agree

bwplotka

Some suggestions

bwplotka · 2018-10-12T11:25:18Z

cmd/thanos/query.go

@@ -87,6 +95,15 @@ func registerQuery(m map[string]setupFunc, app *kingpin.Application, name string
 			lookupStores[s] = struct{}{}
 		}

+		var filesd *file.Discovery


s/filesd/fileSD/ maybe?

bwplotka · 2018-10-12T11:25:47Z

cmd/thanos/query.go

@@ -248,6 +271,14 @@ func runQuery(

 					specs = append(specs, &gossipSpec{id: id, addr: ps.StoreAPIAddr, peer: peer})
 				}
+
+				// Add store specs from file sd.


bwplotka · 2018-10-12T11:28:21Z

cmd/thanos/query.go

+					specs = append(specs, query.NewGRPCStoreSpec(addr))
+				}
+
+				specs = removeDuplicates(specs)


remove or we want to fail maybe?

bwplotka · 2018-10-12T11:28:47Z

cmd/thanos/query.go

@@ -271,6 +302,46 @@ func runQuery(
 			stores.Close()
 		})
 	}
+	// Run File Service Discovery and update the store set when the files are modified.
+	if fileSD != nil {


It is in block, thanks for if (: right?

bwplotka · 2018-10-12T11:32:08Z

cmd/thanos/query.go

+					if update == nil {
+						continue
+					}
+					// TODO(ivan): resolve dns here maybe?


I guess this comment is not needed here? Because further PRs are rdy?

bwplotka · 2018-10-12T11:43:17Z

cmd/thanos/query.go

+			}
+		}, func(error) {
+			cancelUpdate()
+			close(fileSDUpdates)


hm.. Can we ensure we close this ONLY in one place? I feel like both fileSD and this guy closes this, right?

File SD doesn't close it. It is responsibility of the one who passes the channel to file SD.

so why this comment exists?

// Handle the case that a discoverer exits and closes the channel // before the context is done.

yeah, good question. I think this is not needed then. I will remove it. I double checked again but file sd doesn't close this channel.

to clarify, the check when receiving from the channel is not needed.

bwplotka · 2018-10-12T11:43:40Z

cmd/thanos/query.go

@@ -339,6 +410,17 @@ func runQuery(
 	level.Info(logger).Log("msg", "starting query node")
 	return nil
 }
+func removeDuplicates(specs []query.StoreSpec) []query.StoreSpec {


Add newline

log line is must have - if not error

I would really error out

bwplotka · 2018-10-12T11:44:29Z

cmd/thanos/query.go

+	defer f.Unlock()
+	for _, tg := range tgs {
+		// Some Discoverers send nil target group so need to check for it to avoid panics.
+		if tg != nil {


filter out wrong cases to avoid indents and have cleaner code.

Do you mean something like this:

if tg == nil { continue } f.tgs[tg.Source] = tg

bwplotka · 2018-10-12T11:44:51Z

docs/components/query.md

@@ -129,6 +129,10 @@ Flags:
                                 info endpoint (repeated).
      --store=<store> ...        Addresses of statically configured store API
                                 servers (repeatable).
+      --store-sd-file=<path> ...  


could agree

domgreen · 2018-10-12T13:26:58Z

pkg/discovery/cache.go

+}
+
+// NewCache returns a new empty Cache.
+func NewCache() *Cache {


nit - a style preference I have would be to make it pkg/discovery/cache/cache.go which will make it more readable in store.go as it allows cache.New().

domgreen · 2018-10-16T11:22:42Z

cmd/thanos/query.go

+		if len(*fileSDFiles) > 0 {
+			conf := &file.SDConfig{
+				Files:           *fileSDFiles,
+				RefreshInterval: *fileSDInterval,


only thing i can think of here is what if someone does < 1s? or not a valid duration?

Hm, yeah, it is possible to shoot yourself in the foot with this. However, I would argue that if you are using this flag at all you should know what you are doing? What do you think? We can default to 5s if interval < 5s?

Let leave it for the moment, we have a sensible default 👍

domgreen · 2018-10-16T11:43:58Z

cmd/thanos/query.go

+		PlaceHolder("<path>").Strings()
+
+	fileSDInterval := modelDuration(cmd.Flag("store.file-sd-config.interval", "Refresh interval to re-read file SD files. (used as a fallback)").
+		Default("5s"))


Think this should be "5m" 👍

domgreen

Great work on this whole chain of PRs 👍

domgreen · 2018-10-16T15:58:59Z

cmd/thanos/query.go

+		if len(*fileSDFiles) > 0 {
+			conf := &file.SDConfig{
+				Files:           *fileSDFiles,
+				RefreshInterval: *fileSDInterval,


Let leave it for the moment, we have a sensible default 👍

Addressed PR comments More pr comments addressed Added logs and metrics when removing dups Removed redundant check Added flag for file sd refresh interval Changed default fileSD interval to 5m make docs [file-sd-part-2] Query e2e test uses different service discovery options (#554) * Query e2e tests different sd options

ivan-valkov requested a review from bwplotka October 1, 2018 10:20

ivan-valkov force-pushed the file-sd-query branch 3 times, most recently from df34473 to 6c9aca9 Compare October 5, 2018 10:23

ivan-valkov self-assigned this Oct 5, 2018

bwplotka reviewed Oct 5, 2018

View reviewed changes

ivan-valkov force-pushed the file-sd-query branch from 6c9aca9 to 1ff8e29 Compare October 9, 2018 15:30

ivan-valkov changed the base branch from file-sd to master October 9, 2018 15:42

ivan-valkov force-pushed the file-sd-query branch from 1ff8e29 to 2105322 Compare October 9, 2018 16:18

ivan-valkov changed the title ~~Added file sd to query~~ [file-sd-part-1] Added file sd to query Oct 10, 2018

domgreen reviewed Oct 11, 2018

View reviewed changes

bwplotka reviewed Oct 12, 2018

View reviewed changes

bwplotka added feature request/improvement difficulty: medium help wanted labels Oct 12, 2018

domgreen reviewed Oct 12, 2018

View reviewed changes

domgreen reviewed Oct 16, 2018

View reviewed changes

domgreen approved these changes Oct 16, 2018

View reviewed changes

ivan-valkov force-pushed the file-sd-query branch from be86f5c to 97f393c Compare October 16, 2018 16:59

ivan-valkov merged commit 518e851 into master Oct 17, 2018

ivan-valkov deleted the file-sd-query branch October 17, 2018 16:19

[file-sd-part-1] Added file sd to query #546

[file-sd-part-1] Added file sd to query #546

Conversation

ivan-valkov commented Oct 1, 2018 • edited Loading

Changes

Verification

bwplotka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

domgreen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bwplotka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

domgreen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ivan-valkov commented Oct 1, 2018 •

edited

Loading