This was because each client would try to mutate the original set of labels.
Instead of cloning the map which can be expensive, I created a little helper that generate the string
from the labelset without a set of label name.
Fixes grafana#3571
I also added a test to see if this was reproducible and it was indeed:
```
~/go/src/github.com/grafana/loki master*
❯ go test -timeout 30s -tags dev,gofuzz -race -run ^TestMultiClient_Handle_Race$ github.com/grafana/loki/pkg/promtail/client -v -count=1 -timeout=0s
=== RUN TestMultiClient_Handle_Race
==================
WARNING: DATA RACE
Read at 0x00c0000b77d0 by goroutine 22:
runtime.mapaccess2_faststr()
/usr/local/Cellar/go/1.16.2/libexec/src/runtime/map_faststr.go:107 +0x0
github.com/grafana/loki/pkg/promtail/client.(*client).getTenantID()
/Users/ctovena/go/src/github.com/grafana/loki/pkg/promtail/client/client.go:376 +0xcb
github.com/grafana/loki/pkg/promtail/client.(*client).processEntry()
/Users/ctovena/go/src/github.com/grafana/loki/pkg/promtail/client/client.go:407 +0x9a
github.com/grafana/loki/pkg/promtail/client.(*client).run()
/Users/ctovena/go/src/github.com/grafana/loki/pkg/promtail/client/client.go:235 +0x36a
Previous write at 0x00c0000b77d0 by goroutine 21:
runtime.mapdelete_faststr()
/usr/local/Cellar/go/1.16.2/libexec/src/runtime/map_faststr.go:297 +0x0
github.com/grafana/loki/pkg/promtail/client.(*client).processEntry()
/Users/ctovena/go/src/github.com/grafana/loki/pkg/promtail/client/client.go:408 +0x144
github.com/grafana/loki/pkg/promtail/client.(*client).run()
/Users/ctovena/go/src/github.com/grafana/loki/pkg/promtail/client/client.go:235 +0x36a
Goroutine 22 (running) created at:
github.com/grafana/loki/pkg/promtail/client.New()
/Users/ctovena/go/src/github.com/grafana/loki/pkg/promtail/client/client.go:198 +0x7dc
github.com/grafana/loki/pkg/promtail/client.TestMultiClient_Handle_Race()
/Users/ctovena/go/src/github.com/grafana/loki/pkg/promtail/client/multi_test.go:126 +0x2c7
testing.tRunner()
/usr/local/Cellar/go/1.16.2/libexec/src/testing/testing.go:1194 +0x202
Goroutine 21 (running) created at:
github.com/grafana/loki/pkg/promtail/client.New()
/Users/ctovena/go/src/github.com/grafana/loki/pkg/promtail/client/client.go:198 +0x7dc
github.com/grafana/loki/pkg/promtail/client.TestMultiClient_Handle_Race()
/Users/ctovena/go/src/github.com/grafana/loki/pkg/promtail/client/multi_test.go:124 +0x187
testing.tRunner()
/usr/local/Cellar/go/1.16.2/libexec/src/testing/testing.go:1194 +0x202
==================
testing.go:1093: race detected during execution of test
--- FAIL: TestMultiClient_Handle_Race (0.00s)
=== CONT
testing.go:1093: race detected during execution of test
FAIL
FAIL github.com/grafana/loki/pkg/promtail/client 0.022s
FAIL
```
Signed-off-by: Cyril Tovena <cyril.tovena@gmail.com>