Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test failures in i386 #3665

Closed
NightTsarina opened this issue Jan 8, 2018 · 10 comments
Closed

Test failures in i386 #3665

NightTsarina opened this issue Jan 8, 2018 · 10 comments
Labels
component/tests kind/bug priority/P3

Comments

@NightTsarina
Copy link
Contributor

Hi,

Prometheus 2.0 entered Debian a few days ago, and it passes tests correctly in most architectures, except for i386. I checked it is using the exact same versions of packages as amd64.

Looking at the build logs, it seems to me it is an issue with staleness calculations, as I see NaNs where there should be numbers, and viceversa.

The full log is here: https://buildd.debian.org/status/fetch.php?pkg=prometheus&arch=i386&ver=2.0.0%2Bds3-1&stamp=1515102582&raw=0

The relevant errors are as follows:

=== RUN   TestEvaluations
vector result 2 sum(http_requests{job="api-server"}) BY (group)
     {group="canary"} 700 @[3000000]
     {group="production"} 300 @[3000000]

[..]

vector result 4 rate(http_requests{instance!="3"}[1m] OFFSET 10000s)
     {group="canary", instance="0", job="api-server"} 3 @[18000000]
     {group="canary", instance="1", job="api-server"} 4 @[18000000]
     {group="production", instance="0", job="api-server"} 1 @[18000000]
     {group="production", instance="1", job="api-server"} 2 @[18000000]
vector result 4 rate(http_requests[40s]) - rate(http_requests[1m] OFFSET 10000s)
     {group="canary", instance="0", job="api-server"} 5 @[18000000]
     {group="canary", instance="1", job="api-server"} 0 @[18000000]
     {group="production", instance="0", job="api-server"} 2 @[18000000]
     {group="production", instance="1", job="api-server"} 1 @[18000000]
vector result 1 metric
     {__name__="metric"} 1 @[10000]
vector result 1 metric
     {__name__="metric"} NaN @[20000]
--- FAIL: TestEvaluations (1.06s)
	promql_test.go:33: error running test testdata/staleness.test: error in eval metric: unexpected metric {__name__="metric"} in result
FAIL
FAIL	github.com/prometheus/prometheus/promql	1.205s

=== RUN   TestScrapeLoopStop
--- FAIL: TestScrapeLoopStop (0.03s)
	scrape_test.go:376: Appended last sample not as expected. Wanted: stale NaN Got: 7ff8000000000002

=== RUN   TestScrapeLoopRunCreatesStaleMarkersOnFailedScrape
--- FAIL: TestScrapeLoopRunCreatesStaleMarkersOnFailedScrape (0.04s)
	scrape_test.go:526: Appended second sample not as expected. Wanted: stale NaN Got: 7ff8000000000002

=== RUN   TestScrapeLoopRunCreatesStaleMarkersOnParseFailure
--- FAIL: TestScrapeLoopRunCreatesStaleMarkersOnParseFailure (0.02s)
	scrape_test.go:585: Appended second sample not as expected. Wanted: stale NaN Got: 7ff8000000000002

=== RUN   TestScrapeLoopAppendStaleness
--- FAIL: TestScrapeLoopAppendStaleness (0.00s)
	scrape_test.go:699: Appended stale sample wasn't as expected. Wanted: 7ff0000000000002 Got: 7ff8000000000002

=== RUN   TestStaleness
--- FAIL: TestStaleness (0.02s)
	manager_test.go:211: Appended second sample not as expected. Wanted: stale NaN Got: 7ff8000000000002

=== RUN   TestFederation
--- FAIL: TestFederation (0.02s)
	federate_test.go:235: Scenario "empty label value for a label that doesn't exist at all, matches everything": got body
		# TYPE test_metric1 untyped
		test_metric1{foo="bar",instance="i"} 10000 6000000
		test_metric1{foo="boo",instance="i"} 1 6000000
		# TYPE test_metric2 untyped
		test_metric2{foo="boo",instance="i"} 1 6000000
		# TYPE test_metric_old untyped
		test_metric_old{instance=""} 981 5880000
		# TYPE test_metric_stale untyped
		test_metric_stale{instance=""} NaN 6000000
		# TYPE test_metric_without_labels untyped
		test_metric_without_labels{instance=""} 1001 6000000
		
		, want
		# TYPE test_metric1 untyped
		test_metric1{foo="bar",instance="i"} 10000 6000000
		test_metric1{foo="boo",instance="i"} 1 6000000
		# TYPE test_metric2 untyped
		test_metric2{foo="boo",instance="i"} 1 6000000
		# TYPE test_metric_old untyped
		test_metric_old{instance=""} 981 5880000
		# TYPE test_metric_without_labels untyped
		test_metric_without_labels{instance=""} 1001 6000000
		
	federate_test.go:235: Scenario "instance is an external label": got body
		# TYPE test_metric1 untyped
		test_metric1{foo="bar",instance="i"} 10000 6000000
		test_metric1{foo="boo",instance="i"} 1 6000000
		# TYPE test_metric2 untyped
		test_metric2{foo="boo",instance="i"} 1 6000000
		# TYPE test_metric_old untyped
		test_metric_old{instance="baz"} 981 5880000
		# TYPE test_metric_stale untyped
		test_metric_stale{instance="baz"} NaN 6000000
		# TYPE test_metric_without_labels untyped
		test_metric_without_labels{instance="baz"} 1001 6000000
		
		, want
		# TYPE test_metric1 untyped
		test_metric1{foo="bar",instance="i"} 10000 6000000
		test_metric1{foo="boo",instance="i"} 1 6000000
		# TYPE test_metric2 untyped
		test_metric2{foo="boo",instance="i"} 1 6000000
		# TYPE test_metric_old untyped
		test_metric_old{instance="baz"} 981 5880000
		# TYPE test_metric_without_labels untyped
		test_metric_without_labels{instance="baz"} 1001 6000000
		
	federate_test.go:235: Scenario "test_stale_metric": got body
		# TYPE test_metric_stale untyped
		test_metric_stale{instance=""} NaN 6000000
		
		, want
		
	federate_test.go:235: Scenario "everything": got body
		# TYPE test_metric1 untyped
		test_metric1{foo="bar",instance="i"} 10000 6000000
		test_metric1{foo="boo",instance="i"} 1 6000000
		# TYPE test_metric2 untyped
		test_metric2{foo="boo",instance="i"} 1 6000000
		# TYPE test_metric_old untyped
		test_metric_old{instance=""} 981 5880000
		# TYPE test_metric_stale untyped
		test_metric_stale{instance=""} NaN 6000000
		# TYPE test_metric_without_labels untyped
		test_metric_without_labels{instance=""} 1001 6000000
		
		, want
		# TYPE test_metric1 untyped
		test_metric1{foo="bar",instance="i"} 10000 6000000
		test_metric1{foo="boo",instance="i"} 1 6000000
		# TYPE test_metric2 untyped
		test_metric2{foo="boo",instance="i"} 1 6000000
		# TYPE test_metric_old untyped
		test_metric_old{instance=""} 981 5880000
		# TYPE test_metric_without_labels untyped
		test_metric_without_labels{instance=""} 1001 6000000
		
	federate_test.go:235: Scenario "empty label value matches everything that doesn't have that label": got body
		# TYPE test_metric_old untyped
		test_metric_old{instance=""} 981 5880000
		# TYPE test_metric_stale untyped
		test_metric_stale{instance=""} NaN 6000000
		# TYPE test_metric_without_labels untyped
		test_metric_without_labels{instance=""} 1001 6000000
		
		, want
		# TYPE test_metric_old untyped
		test_metric_old{instance=""} 981 5880000
		# TYPE test_metric_without_labels untyped
		test_metric_without_labels{instance=""} 1001 6000000
		
	federate_test.go:235: Scenario "external labels are added if not already present": got body
		# TYPE test_metric1 untyped
		test_metric1{foo="bar",instance="i",zone="ie"} 10000 6000000
		test_metric1{foo="boo",instance="i",zone="ie"} 1 6000000
		# TYPE test_metric2 untyped
		test_metric2{foo="boo",instance="i",zone="ie"} 1 6000000
		# TYPE test_metric_old untyped
		test_metric_old{foo="baz",instance="",zone="ie"} 981 5880000
		# TYPE test_metric_stale untyped
		test_metric_stale{foo="baz",instance="",zone="ie"} NaN 6000000
		# TYPE test_metric_without_labels untyped
		test_metric_without_labels{foo="baz",instance="",zone="ie"} 1001 6000000
		
		, want
		# TYPE test_metric1 untyped
		test_metric1{foo="bar",instance="i",zone="ie"} 10000 6000000
		test_metric1{foo="boo",instance="i",zone="ie"} 1 6000000
		# TYPE test_metric2 untyped
		test_metric2{foo="boo",instance="i",zone="ie"} 1 6000000
		# TYPE test_metric_old untyped
		test_metric_old{foo="baz",instance="",zone="ie"} 981 5880000
		# TYPE test_metric_without_labels untyped
		test_metric_without_labels{foo="baz",instance="",zone="ie"} 1001 6000000
@brian-brazil
Copy link
Contributor

scrape_test.go:699: Appended stale sample wasn't as expected. Wanted: 7ff0000000000002 Got: 7ff8000000000002

I think the CPU is messing with the NaN values.

Do we even support 32bit for 2.0? Given we're mmapping, the address space would be too small.

@NightTsarina
Copy link
Contributor Author

At least it seems to work in other 32 bit arches, like armel: https://buildd.debian.org/status/package.php?p=prometheus

@brian-brazil
Copy link
Contributor

I'd suspect a floating point issue here, something is changing the signalling NaN to a quiet NaN.

Prometheus isn't going to be very useful on a 32bit machine.

@NightTsarina
Copy link
Contributor Author

If you guys decide to stop supporting 32 bits at all, please let me know, as I should request removal from the archive for all these arches.

@brian-brazil
Copy link
Contributor

@fabxc We don't support 2.0 for 32-bit, right?

@fabxc
Copy link
Contributor

fabxc commented Jan 8, 2018 via email

@NightTsarina
Copy link
Contributor Author

Well, at least for i386 it is not working at all. I will ask the Debian ftp-masters to remove that arch.

In any case, are you guys sure this is not a hidden bug somewhere?

@brian-brazil
Copy link
Contributor

I would guess that the value is being passed through the FPU at some point, which it shouldn't be.

@beorn7
Copy link
Member

beorn7 commented Feb 27, 2024

Hello from the bug scrub.

We doubt that anyone will ever look into this issue, so we are closing it. As a side note, it would be interesting if this test failure actually still happens. (Maybe it was a problem in the Go runtime or we changed the relevant code in the last 6 years?)

@beorn7 beorn7 closed this as completed Feb 27, 2024
@NightTsarina
Copy link
Contributor Author

@beorn7 I had forgotten about this... I checked and it seems that i386 started passing tests again at the end of 2021, when we uploaded 2.31 (logs here), so this issue was solved somehow :)

@prometheus prometheus locked as resolved and limited conversation to collaborators Aug 25, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
component/tests kind/bug priority/P3
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@simonpasquier @fabxc @NightTsarina @beorn7 @brian-brazil