Unexpected order dependency in promtool test rules unit tests

[vatine]

Bug Report

What did you do?
Wrote a rule unit-test

What did you expect to see?
The unit-test passing, without having to specify my test cases in any specific order (it looks like the exp_samples need to have the labels in lexicographic order, which is quite unexpected).

What did you see instead? Under which circumstances?
The unit-test pass/fail depends on the ordering of the exp_samples, a minimal test case is attached.

Environment
MacOS and Linux

• System information:

  insert output of uname -srm here

• Prometheus version:

  insert output of prometheus --version here

• Alertmanager version:

  insert output of alertmanager --version here (if relevant to the issue)

• Prometheus configuration file:

groups:
  - name: test
    rules:
      - record: job:testvalue_by_foo:sum
        expr: sum(testvalue) by (foo)

• Passing unit test

rule_files:
  - 'rules.yaml'
tests:
  - interval: '1m'
    input_series:
      - series: 'testvalue{foo="a", bar="a"}'
        values: '1+0x30'
      - series: 'testvalue{foo="a", bar="b"}'
        values: '1+0x30'
      - series: 'testvalue{foo="b", bar="a"}'
        values: '1+0x30'
    promql_expr_test:
      - expr: 'job:testvalue_by_foo:sum'
        eval_time: 3m
        exp_samples:
          - labels: 'job:testvalue_by_foo:sum{foo="a"}'
            value: 2
          - labels: 'job:testvalue_by_foo:sum{foo="b"}'
            value: 1

Failing unit tests:

rule_files:
  - 'rules.yaml'
tests:
  - interval: '1m'
    input_series:
      - series: 'testvalue{foo="a", bar="a"}'
        values: '1+0x30'
      - series: 'testvalue{foo="a", bar="b"}'
        values: '1+0x30'
      - series: 'testvalue{foo="b", bar="a"}'
        values: '1+0x30'
    promql_expr_test:
      - expr: 'job:testvalue_by_foo:sum'
        eval_time: 3m
        exp_samples:
          - labels: 'job:testvalue_by_foo:sum{foo="b"}'
            value: 1
          - labels: 'job:testvalue_by_foo:sum{foo="a"}'
            value: 2

Example output:

Leabhar:prometheus-fail ingvar$ ~/Downloads/prometheus-2.6.0.darwin-amd64/promtool test rules test1.yaml 
Unit Testing:  test1.yaml
  SUCCESS

Leabhar:prometheus-fail ingvar$ ~/Downloads/prometheus-2.6.0.darwin-amd64/promtool test rules test2.yaml 
Unit Testing:  test2.yaml
  FAILED:
    expr:'job:testvalue_by_foo:sum', time:3m0s, 
        exp:"{__name__=\"job:testvalue_by_foo:sum\", foo=\"b\"} 1E+00, {__name__=\"job:testvalue_by_foo:sum\", foo=\"b\"} 1E+00, {__name__=\"job:testvalue_by_foo:sum\", foo=\"a\"} 2E+00", 
        got:"{__name__=\"job:testvalue_by_foo:sum\", foo=\"a\"} 2E+00, {__name__=\"job:testvalue_by_foo:sum\", foo=\"a\"} 2E+00, {__name__=\"job:testvalue_by_foo:sum\", foo=\"b\"} 1E+00"

Leabhar:prometheus-fail ingvar$ ~/Downloads/prometheus-2.6.0.darwin-amd64/promtool --version
promtool, version 2.6.0 (branch: HEAD, revision: dbd1d58c894775c0788470944b818cc724f550fb)
  build user:       root@bf5760470f13
  build date:       20181217-15:19:45
  go version:       go1.11.3

All this with the official MacOS build, but initially found on Linux.

[juliusv]

/cc @codesome

I don't see https://prometheus.io/docs/prometheus/latest/configuration/unit_testing_rules/#promql_test_case mentioning ordering requirements, so this seems like a bug.

[codesome]

Will take a look in couple of days, thanks for reporting @vatine.

[codesome]

Yes this is a bug, I think I assumed the order when implementing it. I will open a PR with the fix.