Ordering of targets in a group is not consistent

[martinhanzik]

Bug Report

What did you do?
I added a file scraper using swarm-discovery that automatically generates the file from the docker API.
I opened the 'Targets' page.

What did you expect to see?
Targets in a consistent order, ordered by job first, then label, as specified by #4898

What did you see instead? Under which circumstances?
When refreshing the page, the target order in the group changes. Targets for one job seem to always be ordered by instance, but they mix with the other jobs.

I don't know why https://github.com/prometheus/prometheus/blob/v2.7.0/web/web.go#L698 is using the OR operation for the 2 conditions, normally one would do something like this:

if (a.L1 == a.L1)
  return a.L2 < b.L2;
return a.L1 < b.L1;

Environment

• Prometheus version:

  2.7.0

• Prometheus configuration file:

scrape_configs:
  - job_name: swarm-service-endpoints
    file_sd_configs:
      - files:
        - /etc/swarm-endpoints/swarm-endpoints.json
    # node exporter + container exporter
    metric_relabel_configs:
    # Set the instance to the hostname
    - source_labels: [__meta_swarm_node_hostname]
      target_label: instance
    # scrap only running tasks
    - source_labels: [__meta_swarm_task_desired_state]
      regex: running
      action: keep
    # Add a task label
    - source_labels: [__meta_swarm_task_name]
      target_label: task

[martinhanzik]

I forgot to provide the labels for this problem:

• instance=swarm1, job=A
• instance=swarm2, job=A
• instance=swarm3, job=A
• instance=swarm1, job=B
• instance=swarm2, job=B
• instance=swarm3, job=B
• instance=swarm1, job=C

[yeya24]

Yeah the logic is a little bit wrong. But I can't reproduce this error. How to reproduce it? Here in my webui the targets are always ordered

[codesome]

If the labels come sorted with instance first and then job, then it will stay like that and won't be sorted by job first.

[codesome]

Opening a PR with the fix in a bit.

[cstyan]

@codesome looks like this can be closed?

[martinhanzik]

Looks that way