Skip to content

/prometheus

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test failures in i386 #3665

Open
TheTincho opened this Issue Jan 8, 2018 · 8 comments

Comments

Assignees
No one assigned
Labels
component/tests kind/question
Projects
None yet
Milestone
No milestone
4 participants
@TheTincho @brian-brazil @fabxc @simonpasquier
@TheTincho
Copy link
Contributor

TheTincho commented Jan 8, 2018

Hi,

Prometheus 2.0 entered Debian a few days ago, and it passes tests correctly in most architectures, except for i386. I checked it is using the exact same versions of packages as amd64.

Looking at the build logs, it seems to me it is an issue with staleness calculations, as I see NaNs where there should be numbers, and viceversa.

The full log is here: https://buildd.debian.org/status/fetch.php?pkg=prometheus&arch=i386&ver=2.0.0%2Bds3-1&stamp=1515102582&raw=0

The relevant errors are as follows:

=== RUN   TestEvaluations
vector result 2 sum(http_requests{job="api-server"}) BY (group)
     {group="canary"} 700 @[3000000]
     {group="production"} 300 @[3000000]

[..]

vector result 4 rate(http_requests{instance!="3"}[1m] OFFSET 10000s)
     {group="canary", instance="0", job="api-server"} 3 @[18000000]
     {group="canary", instance="1", job="api-server"} 4 @[18000000]
     {group="production", instance="0", job="api-server"} 1 @[18000000]
     {group="production", instance="1", job="api-server"} 2 @[18000000]
vector result 4 rate(http_requests[40s]) - rate(http_requests[1m] OFFSET 10000s)
     {group="canary", instance="0", job="api-server"} 5 @[18000000]
     {group="canary", instance="1", job="api-server"} 0 @[18000000]
     {group="production", instance="0", job="api-server"} 2 @[18000000]
     {group="production", instance="1", job="api-server"} 1 @[18000000]
vector result 1 metric
     {__name__="metric"} 1 @[10000]
vector result 1 metric
     {__name__="metric"} NaN @[20000]
--- FAIL: TestEvaluations (1.06s)
	promql_test.go:33: error running test testdata/staleness.test: error in eval metric: unexpected metric {__name__="metric"} in result
FAIL
FAIL	github.com/prometheus/prometheus/promql	1.205s

=== RUN   TestScrapeLoopStop
--- FAIL: TestScrapeLoopStop (0.03s)
	scrape_test.go:376: Appended last sample not as expected. Wanted: stale NaN Got: 7ff8000000000002

=== RUN   TestScrapeLoopRunCreatesStaleMarkersOnFailedScrape
--- FAIL: TestScrapeLoopRunCreatesStaleMarkersOnFailedScrape (0.04s)
	scrape_test.go:526: Appended second sample not as expected. Wanted: stale NaN Got: 7ff8000000000002

=== RUN   TestScrapeLoopRunCreatesStaleMarkersOnParseFailure
--- FAIL: TestScrapeLoopRunCreatesStaleMarkersOnParseFailure (0.02s)
	scrape_test.go:585: Appended second sample not as expected. Wanted: stale NaN Got: 7ff8000000000002

=== RUN   TestScrapeLoopAppendStaleness
--- FAIL: TestScrapeLoopAppendStaleness (0.00s)
	scrape_test.go:699: Appended stale sample wasn't as expected. Wanted: 7ff0000000000002 Got: 7ff8000000000002

=== RUN   TestStaleness
--- FAIL: TestStaleness (0.02s)
	manager_test.go:211: Appended second sample not as expected. Wanted: stale NaN Got: 7ff8000000000002

=== RUN   TestFederation
--- FAIL: TestFederation (0.02s)
	federate_test.go:235: Scenario "empty label value for a label that doesn't exist at all, matches everything": got body
		# TYPE test_metric1 untyped
		test_metric1{foo="bar",instance="i"} 10000 6000000
		test_metric1{foo="boo",instance="i"} 1 6000000
		# TYPE test_metric2 untyped
		test_metric2{foo="boo",instance="i"} 1 6000000
		# TYPE test_metric_old untyped
		test_metric_old{instance=""} 981 5880000
		# TYPE test_metric_stale untyped
		test_metric_stale{instance=""} NaN 6000000
		# TYPE test_metric_without_labels untyped
		test_metric_without_labels{instance=""} 1001 6000000
		
		, want
		# TYPE test_metric1 untyped
		test_metric1{foo="bar",instance="i"} 10000 6000000
		test_metric1{foo="boo",instance="i"} 1 6000000
		# TYPE test_metric2 untyped
		test_metric2{foo="boo",instance="i"} 1 6000000
		# TYPE test_metric_old untyped
		test_metric_old{instance=""} 981 5880000
		# TYPE test_metric_without_labels untyped
		test_metric_without_labels{instance=""} 1001 6000000
		
	federate_test.go:235: Scenario "instance is an external label": got body
		# TYPE test_metric1 untyped
		test_metric1{foo="bar",instance="i"} 10000 6000000
		test_metric1{foo="boo",instance="i"} 1 6000000
		# TYPE test_metric2 untyped
		test_metric2{foo="boo",instance="i"} 1 6000000
		# TYPE test_metric_old untyped
		test_metric_old{instance="baz"} 981 5880000
		# TYPE test_metric_stale untyped
		test_metric_stale{instance="baz"} NaN 6000000
		# TYPE test_metric_without_labels untyped
		test_metric_without_labels{instance="baz"} 1001 6000000
		
		, want
		# TYPE test_metric1 untyped
		test_metric1{foo="bar",instance="i"} 10000 6000000
		test_metric1{foo="boo",instance="i"} 1 6000000
		# TYPE test_metric2 untyped
		test_metric2{foo="boo",instance="i"} 1 6000000
		# TYPE test_metric_old untyped
		test_metric_old{instance="baz"} 981 5880000
		# TYPE test_metric_without_labels untyped
		test_metric_without_labels{instance="baz"} 1001 6000000
		
	federate_test.go:235: Scenario "test_stale_metric": got body
		# TYPE test_metric_stale untyped
		test_metric_stale{instance=""} NaN 6000000
		
		, want
		
	federate_test.go:235: Scenario "everything": got body
		# TYPE test_metric1 untyped
		test_metric1{foo="bar",instance="i"} 10000 6000000
		test_metric1{foo="boo",instance="i"} 1 6000000
		# TYPE test_metric2 untyped
		test_metric2{foo="boo",instance="i"} 1 6000000
		# TYPE test_metric_old untyped
		test_metric_old{instance=""} 981 5880000
		# TYPE test_metric_stale untyped
		test_metric_stale{instance=""} NaN 6000000
		# TYPE test_metric_without_labels untyped
		test_metric_without_labels{instance=""} 1001 6000000
		
		, want
		# TYPE test_metric1 untyped
		test_metric1{foo="bar",instance="i"} 10000 6000000
		test_metric1{foo="boo",instance="i"} 1 6000000
		# TYPE test_metric2 untyped
		test_metric2{foo="boo",instance="i"} 1 6000000
		# TYPE test_metric_old untyped
		test_metric_old{instance=""} 981 5880000
		# TYPE test_metric_without_labels untyped
		test_metric_without_labels{instance=""} 1001 6000000
		
	federate_test.go:235: Scenario "empty label value matches everything that doesn't have that label": got body
		# TYPE test_metric_old untyped
		test_metric_old{instance=""} 981 5880000
		# TYPE test_metric_stale untyped
		test_metric_stale{instance=""} NaN 6000000
		# TYPE test_metric_without_labels untyped
		test_metric_without_labels{instance=""} 1001 6000000
		
		, want
		# TYPE test_metric_old untyped
		test_metric_old{instance=""} 981 5880000
		# TYPE test_metric_without_labels untyped
		test_metric_without_labels{instance=""} 1001 6000000
		
	federate_test.go:235: Scenario "external labels are added if not already present": got body
		# TYPE test_metric1 untyped
		test_metric1{foo="bar",instance="i",zone="ie"} 10000 6000000
		test_metric1{foo="boo",instance="i",zone="ie"} 1 6000000
		# TYPE test_metric2 untyped
		test_metric2{foo="boo",instance="i",zone="ie"} 1 6000000
		# TYPE test_metric_old untyped
		test_metric_old{foo="baz",instance="",zone="ie"} 981 5880000
		# TYPE test_metric_stale untyped
		test_metric_stale{foo="baz",instance="",zone="ie"} NaN 6000000
		# TYPE test_metric_without_labels untyped
		test_metric_without_labels{foo="baz",instance="",zone="ie"} 1001 6000000
		
		, want
		# TYPE test_metric1 untyped
		test_metric1{foo="bar",instance="i",zone="ie"} 10000 6000000
		test_metric1{foo="boo",instance="i",zone="ie"} 1 6000000
		# TYPE test_metric2 untyped
		test_metric2{foo="boo",instance="i",zone="ie"} 1 6000000
		# TYPE test_metric_old untyped
		test_metric_old{foo="baz",instance="",zone="ie"} 981 5880000
		# TYPE test_metric_without_labels untyped
		test_metric_without_labels{foo="baz",instance="",zone="ie"} 1001 6000000
@brian-brazil

This comment has been minimized.

Sign in to view
Copy link
Member

brian-brazil commented Jan 8, 2018

scrape_test.go:699: Appended stale sample wasn't as expected. Wanted: 7ff0000000000002 Got: 7ff8000000000002

I think the CPU is messing with the NaN values.

Do we even support 32bit for 2.0? Given we're mmapping, the address space would be too small.
@TheTincho

This comment has been minimized.

Sign in to view
Copy link
Contributor Author

TheTincho commented Jan 8, 2018

At least it seems to work in other 32 bit arches, like armel: https://buildd.debian.org/status/package.php?p=prometheus
@brian-brazil

This comment has been minimized.

Sign in to view
Copy link
Member

brian-brazil commented Jan 8, 2018

I'd suspect a floating point issue here, something is changing the signalling NaN to a quiet NaN.

Prometheus isn't going to be very useful on a 32bit machine.
@TheTincho

This comment has been minimized.

Sign in to view
Copy link
Contributor Author

TheTincho commented Jan 8, 2018

If you guys decide to stop supporting 32 bits at all, please let me know, as I should request removal from the archive for all these arches.
@brian-brazil

This comment has been minimized.

Sign in to view
Copy link
Member

brian-brazil commented Jan 8, 2018

@fabxc We don't support 2.0 for 32-bit, right?
@fabxc

This comment has been minimized.

Sign in to view
Copy link
Member

fabxc commented Jan 8, 2018

@TheTincho

This comment has been minimized.

Sign in to view
Copy link
Contributor Author

TheTincho commented Jan 9, 2018

Well, at least for i386 it is not working at all. I will ask the Debian ftp-masters to remove that arch.

In any case, are you guys sure this is not a hidden bug somewhere?
@brian-brazil

This comment has been minimized.

Sign in to view
Copy link
Member

brian-brazil commented Jan 9, 2018

I would guess that the value is being passed through the FPU at some point, which it shouldn't be.

@simonpasquier simonpasquier added kind/question component/tests labels Sep 7, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.