[FLAKE]make test fails on osx and windows in travis CI #2570

prietyc123 · 2020-02-06T07:39:49Z

/kind bug
/kind flake

What versions of software are you using?

Operating System:

Output of odo version: master

How did you run odo exactly?

On OpenShift CI

Actual behavior

Failed in getting the details of the changed file TestWatchAndPush on macOS.

Also watch.go is creating some temporary files C:\Users\travis\AppData\Local\Temp\fabric8-analytics-license-analysis047095974\tests417416904\test_read_licenses.py216365071 which is failed to get detected on windows.

Expected behavior

make test should pass

Any logs, error output, etc?

macOS travis CI

$ make test
go test -race github.com/openshift/odo/cmd/cli-doc github.com/openshift/odo/cmd/odo github.com/openshift/odo/pkg/application github.com/openshift/odo/pkg/application/labels github.com/openshift/odo/pkg/auth github.com/openshift/odo/pkg/catalog github.com/openshift/odo/pkg/component github.com/openshift/odo/pkg/component/labels github.com/openshift/odo/pkg/config 
[...]
ok  	github.com/openshift/odo/pkg/catalog	1.280s
E0131 09:18:12.583395    5459 watch.go:183] Failed getting details of the changed file /var/folders/nz/vv4_9tw56nv9k3tkvyszvwg80000gn/T/fabric8-analytics-license-analysis259913361/tests868419307/test_read_licenses.py612824246. Ignoring the change
E0131 09:18:12.585345    5459 watch.go:189] Ignoring event for file /var/folders/nz/vv4_9tw56nv9k3tkvyszvwg80000gn/T/fabric8-analytics-license-analysis259913361/tests868419307/test_read_licenses.py612824246 as details about the file couldn't be fetched
--- FAIL: TestWatchAndPush (11.10s)
    watch_test.go:690: Running test:  Case 1: Valid watch with list of files to be ignored with a append event
    watch_test.go:690: Running test:  Case 2: Valid watch with list of files to be ignored with a append and a delete event
    watch_test.go:690: Running test:  Case 3: Valid watch with list of files to be ignored with a create and a delete event
    watch_test.go:690: Running test:  Case 4: Valid watch with list of files to be ignored with a folder create event
    --- FAIL: TestWatchAndPush/Case_4:_Valid_watch_with_list_of_files_to_be_ignored_with_a_folder_create_event (1.03s)
 [...]

windows travis CI

$ make test
go test  -race github.com/openshift/odo/cmd/cli-doc github.com/openshift/odo/cmd/odo github.com/openshift/odo/pkg/application github.com/openshift/odo/pkg/application/labels github.com/openshift/odo/pkg/auth github.com/openshift/odo/pkg/catalog github.com/openshift/odo/pkg/component github.com/openshift/odo/pkg/component/labels github.com/openshift/odo/pkg/config github.com/openshift/odo/pkg/debug github.com/openshift/odo/pkg/devfile github.com/openshift/odo/pkg/devfile/adapters
[...]
E0303 10:03:25.148080    5044 watch.go:183] Failed getting details of the changed file C:\Users\travis\AppData\Local\Temp\fabric8-analytics-license-analysis047095974\tests417416904\test_read_licenses.py216365071. Ignoring the change
windows failed err CreateFile C:\Users\travis\AppData\Local\Temp\fabric8-analytics-license-analysis047095974\tests417416904\test_read_licenses.py216365071: The system cannot find the path specified.
E0303 10:03:25.152962    5044 watch.go:190] Ignoring event for file C:\Users\travis\AppData\Local\Temp\fabric8-analytics-license-analysis047095974\tests417416904\test_read_licenses.py216365071 as details about the file couldn't be fetched
windows ignore stat err CreateFile C:\Users\travis\AppData\Local\Temp\fabric8-analytics-license-analysis047095974\tests417416904\test_read_licenses.py216365071: The system cannot find the path specified.
E0303 10:03:25.153939    5044 watch.go:183] Failed getting details of the changed file C:\Users\travis\AppData\Local\Temp\fabric8-analytics-license-analysis047095974\tests417416904. Ignoring the change
windows failed err CreateFile C:\Users\travis\AppData\Local\Temp\fabric8-analytics-license-analysis047095974\tests417416904: The system cannot find the file specified.
E0303 10:03:25.153939    5044 watch.go:190] Ignoring event for file C:\Users\travis\AppData\Local\Temp\fabric8-analytics-license-analysis047095974\tests417416904 as details about the file couldn't be fetched
windows ignore stat err CreateFile C:\Users\travis\AppData\Local\Temp\fabric8-analytics-license-analysis047095974\tests417416904: The system cannot find the file specified.
E0303 10:03:25.153939    5044 watch.go:183] Failed getting details of the changed file C:\Users\travis\AppData\Local\Temp\fabric8-analytics-license-analysis047095974\tests417416904. Ignoring the change
windows failed err CreateFile C:\Users\travis\AppData\Local\Temp\fabric8-analytics-license-analysis047095974\tests417416904: The system cannot find the file specified.
E0303 10:03:25.153939    5044 watch.go:190] Ignoring event for file C:\Users\travis\AppData\Local\Temp\fabric8-analytics-license-analysis047095974\tests417416904 as details about the file couldn't be fetched
windows ignore stat err CreateFile C:\Users\travis\AppData\Local\Temp\fabric8-analytics-license-analysis047095974\tests417416904: The system cannot find the file specified.
received [C:\Users\travis\AppData\Local\Temp\fabric8-analytics-license-analysis047095974\__init__.py011991377 C:\Users\travis\AppData\Local\Temp\fabric8-analytics-license-analysis047095974\src445007821\read_licenses.py313912176] which is not same as expected list src/read_licenses.py,__init__.pyFAIL	github.com/openshift/odo/pkg/component	2.452s
ok  	github.com/openshift/odo/pkg/component/labels	1.047s
ok  	github.com/openshift/odo/pkg/config	1.147s
--- FAIL: Test_getDebugInfo (0.01s)
    --- FAIL: Test_getDebugInfo/case_3:_debug_port_not_listening (0.00s)
        info_test.go:315: getDebugInfo() got = {{OdoDebugInfo v1} 3844 testing-1 app nodejs-ex 5858 9001}, want {{ } 0    0 0}
        info_test.go:318: getDebugInfo() got1 = true, want false
    --- FAIL: Test_getDebugInfo/case_4:_the_process_is_not_running (0.00s)
        info_test.go:302: error while starting fake port listerner, cause: listen tcp 127.0.0.1:9001: bind: Only one usage of each socket address (protocol/network address/port) is normally permitted.
FAIL
FAIL	github.com/openshift/odo/pkg/debug	0.377s
[...]

Issue ref : https://travis-ci.com/openshift/odo/jobs/281881842#L133 and https://travis-ci.com/openshift/odo/jobs/293340097#L110

The text was updated successfully, but these errors were encountered:

amitkrout · 2020-02-06T09:34:02Z

@mik-dass Based on your comment #2571 (comment) on the duplicate issue #2571, do we really need this issue ? cc_ @prietyc123

mik-dass · 2020-02-06T09:43:46Z

@mik-dass Based on your comment #2571 (comment) on the duplicate issue #2571, do we really need this issue ? cc_ @prietyc123

Yes this is a different issue than that. This was introduced in #2549 (comment). This was the reason we bumped the go version on the travis CI.

kadel · 2020-02-13T17:54:52Z

/area testing

kadel · 2020-02-20T14:23:33Z

/priority high

prietyc123 · 2020-03-06T09:49:31Z

Skipped watch test on windows and macOS in https://github.com/openshift/odo/pull/2562/files#r389347456. @girishramnani

mik-dass · 2020-03-16T11:42:04Z

#2692 only solved the flake for windows.

amitkrout · 2020-03-29T10:55:03Z

@mik-dass it seems the flake on mac (travis CI) appears more frequently, even i can now reproduce it locally on my mac setup.

occurrence - https://travis-ci.com/github/openshift/odo/jobs/307573205#L210

mik-dass · 2020-04-13T07:42:17Z

Any progress on this?

No

This is a high priority, flakes are slowing us down a lot!

Since this test passes on Linux and windows, let's keep the issue open and skip it for mac OS for now. WDYT?

kadel · 2020-04-14T10:05:36Z

Since this test passes on Linux and windows, let's keep the issue open and skip it for mac OS for now. WDYT?

Let's try to fix it. I don't think that skipping tests is a good idea.

kadel · 2020-04-14T12:45:52Z

occurrence: https://travis-ci.com/github/openshift/odo/jobs/317396190

kadel · 2020-04-15T11:29:06Z

https://travis-ci.com/github/openshift/odo/jobs/317396190

mik-dass · 2020-04-15T15:35:11Z

On linux

[mrinaldas@localhost odo]$ go test pkg/watch/watch.go pkg/watch/watch_test.go -p 5 -count=1 -v 4
=== RUN   TestWatchAndPush
=== RUN   TestWatchAndPush/Case_1:_Valid_watch_with_list_of_files_to_be_ignored_with_a_append_event
Waiting for something to change in /tmp/fabric8-analytics-license-analysis858884704
=== RUN   TestWatchAndPush/Case_2:_Valid_watch_with_list_of_files_to_be_ignored_with_a_append_and_a_delete_event
Waiting for something to change in /tmp/fabric8-analytics-license-analysis082560444
=== RUN   TestWatchAndPush/Case_3:_Valid_watch_with_list_of_files_to_be_ignored_with_a_create_and_a_delete_event
Waiting for something to change in /tmp/fabric8-analytics-license-analysis409193560
E0415 20:47:38.141679   15000 watch.go:186] Failed getting details of the changed file /tmp/fabric8-analytics-license-analysis409193560/tests463270730/test_read_licenses.py613216233. Ignoring the change
E0415 20:47:38.142007   15000 watch.go:192] Ignoring event for file /tmp/fabric8-analytics-license-analysis409193560/tests463270730/test_read_licenses.py613216233 as details about the file couldn't be fetched
=== RUN   TestWatchAndPush/Case_4:_Valid_watch_with_list_of_files_to_be_ignored_with_a_folder_create_event
Waiting for something to change in /tmp/fabric8-analytics-license-analysis901342260
E0415 20:47:41.145053   15000 watch.go:186] Failed getting details of the changed file /tmp/fabric8-analytics-license-analysis901342260/tests235530630/test_read_licenses.py381033589. Ignoring the change
E0415 20:47:41.145104   15000 watch.go:192] Ignoring event for file /tmp/fabric8-analytics-license-analysis901342260/tests235530630/test_read_licenses.py381033589 as details about the file couldn't be fetched
=== RUN   TestWatchAndPush/Case_5:_Valid_watch_with_list_of_files_to_be_ignored_with_a_folder_delete_event
Waiting for something to change in /tmp/fabric8-analytics-license-analysis054912111
E0415 20:47:44.152919   15000 watch.go:186] Failed getting details of the changed file /tmp/fabric8-analytics-license-analysis054912111/tests653345401/test_read_licenses.py735065371. Ignoring the change
E0415 20:47:44.152983   15000 watch.go:192] Ignoring event for file /tmp/fabric8-analytics-license-analysis054912111/tests653345401/test_read_licenses.py735065371 as details about the file couldn't be fetched
--- PASS: TestWatchAndPush (14.03s)
    --- PASS: TestWatchAndPush/Case_1:_Valid_watch_with_list_of_files_to_be_ignored_with_a_append_event (2.01s)
    --- PASS: TestWatchAndPush/Case_2:_Valid_watch_with_list_of_files_to_be_ignored_with_a_append_and_a_delete_event (3.01s)
    --- PASS: TestWatchAndPush/Case_3:_Valid_watch_with_list_of_files_to_be_ignored_with_a_create_and_a_delete_event (3.00s)
    --- PASS: TestWatchAndPush/Case_4:_Valid_watch_with_list_of_files_to_be_ignored_with_a_folder_create_event (3.01s)
    --- PASS: TestWatchAndPush/Case_5:_Valid_watch_with_list_of_files_to_be_ignored_with_a_folder_delete_event (3.01s)
PASS
ok      command-line-arguments  14.044s



[mrinaldas@localhost odo]$ go test pkg/watch/watch.go pkg/watch/watch_test.go -p 1 -count=1 -v 4
=== RUN   TestWatchAndPush
=== RUN   TestWatchAndPush/Case_1:_Valid_watch_with_list_of_files_to_be_ignored_with_a_append_event
Waiting for something to change in /tmp/fabric8-analytics-license-analysis893651643
=== RUN   TestWatchAndPush/Case_2:_Valid_watch_with_list_of_files_to_be_ignored_with_a_append_and_a_delete_event
Waiting for something to change in /tmp/fabric8-analytics-license-analysis602517927
=== RUN   TestWatchAndPush/Case_3:_Valid_watch_with_list_of_files_to_be_ignored_with_a_create_and_a_delete_event
Waiting for something to change in /tmp/fabric8-analytics-license-analysis760146131
E0415 20:48:16.371745   15088 watch.go:186] Failed getting details of the changed file /tmp/fabric8-analytics-license-analysis760146131/tests268043901/test_read_licenses.py632036960. Ignoring the change
E0415 20:48:16.372775   15088 watch.go:192] Ignoring event for file /tmp/fabric8-analytics-license-analysis760146131/tests268043901/test_read_licenses.py632036960 as details about the file couldn't be fetched
=== RUN   TestWatchAndPush/Case_4:_Valid_watch_with_list_of_files_to_be_ignored_with_a_folder_create_event
Waiting for something to change in /tmp/fabric8-analytics-license-analysis354528831
=== RUN   TestWatchAndPush/Case_5:_Valid_watch_with_list_of_files_to_be_ignored_with_a_folder_delete_event
Waiting for something to change in /tmp/fabric8-analytics-license-analysis675753038
--- PASS: TestWatchAndPush (14.02s)
    --- PASS: TestWatchAndPush/Case_1:_Valid_watch_with_list_of_files_to_be_ignored_with_a_append_event (2.00s)
    --- PASS: TestWatchAndPush/Case_2:_Valid_watch_with_list_of_files_to_be_ignored_with_a_append_and_a_delete_event (3.00s)
    --- PASS: TestWatchAndPush/Case_3:_Valid_watch_with_list_of_files_to_be_ignored_with_a_create_and_a_delete_event (3.00s)
    --- PASS: TestWatchAndPush/Case_4:_Valid_watch_with_list_of_files_to_be_ignored_with_a_folder_create_event (3.01s)
    --- PASS: TestWatchAndPush/Case_5:_Valid_watch_with_list_of_files_to_be_ignored_with_a_folder_delete_event (3.01s)
PASS
ok      command-line-arguments  14.040s

Increasing the parallel nodes on the watch test to 5 seems to result in 3 Failed getting details of the changed file cases instead of 1 case

prietyc123 · 2020-05-20T09:28:32Z

@mik-dass We are hitting the same issue on windows https://travis-ci.com/github/openshift/odo/jobs/337120311#L233

mik-dass · 2020-05-20T09:58:25Z

@mik-dass We are hitting the same issue on windows https://travis-ci.com/github/openshift/odo/jobs/337120311#L233

 watch_test.go:823: error in WatchAndPush error watching filesystem for changes: Windows system assumed buffer larger than it is, events have likely been missed.

It was thrown by the fsnotify library https://github.com/fsnotify/fsnotify/blob/master/windows.go#L506 due to some problem with the windows OS.

prietyc123 · 2020-05-20T10:19:08Z

It was thrown by the fsnotify library https://github.com/fsnotify/fsnotify/blob/master/windows.go#L506 due to some problem with the windows OS.

Ok, then we should add patch in the upstream or let’s create an issue there to keep a track of that. Or is there any possibility to fix it from our end ?

amitkrout · 2020-05-20T10:25:15Z

Hitting the same issue again - https://travis-ci.com/github/openshift/odo/jobs/337411640#L248

pr reference - #3218

mik-dass · 2020-05-20T10:25:27Z

Ok, then we should add patch in the upstream or let’s create an issue there to keep a track of that. Or is there any possibility to fix it from our end ?

The error is already handled upstream by throwing an error from their side. To fix this, we probably will need fix from windows side. The only other solution I can think of is ignoring this error in code. But that can lead to other problems.

amitkrout · 2020-05-20T11:03:29Z

@mik-dass Skip is not a good idea as you mentioned. This issue has been hit twice, and i am sure more will come in recent future. AFAIK travis does not do image update, then how come suddenly our test fails. Or am i missing anything ?

amitkrout · 2020-05-20T11:06:31Z

one more hit - https://travis-ci.com/github/openshift/odo/jobs/337411640#L248

mik-dass · 2020-05-20T11:19:37Z

@mik-dass Skip is not a good idea as you mentioned. This issue has been hit twice, and i am sure more will come in recent future.

Yeah more will come and I am sure about that too.

AFAIK travis does not do image update, then how come suddenly our test fails?

Are you sure about that?

amitkrout · 2020-05-20T11:27:15Z

@mik-dass Skip is not a good idea as you mentioned. This issue has been hit twice, and i am sure more will come in recent future.

Yeah more will come and I am sure about that too.

AFAIK travis does not do image update, then how come suddenly our test fails?

Are you sure about that?

yes, just for your reference i am attaching a screen shoot. Logs collected from the travis CI

23rd Mar ------------------------------------------- Today

mik-dass · 2020-05-20T12:20:10Z

yes, just for your reference i am attaching a screen shoot. Logs collected from the travis CI

Ok let me check locally on a windows machine

mik-dass · 2020-05-20T17:41:35Z

Ok let me check locally on a windows machine

I have tested the latest master watch test code 100+ times on a windows machine but couldn't reproduce the issue a single time. Also I think we should close this issue and create a new one as, log wise, most probably it has nothing to do with the issue reported.

amitkrout · 2020-05-21T05:44:40Z

Ok let me check locally on a windows machine

I have tested the latest master watch test code 100+ times on a windows machine but couldn't reproduce the issue a single time. Also I think we should close this issue and create a new one as, log wise, most probably it has nothing to do with the issue reported.

Ok, make sense. @prietyc123 Please go ahead and open an issue with the fresh logs.

prietyc123 · 2020-05-21T06:15:34Z

Ok, make sense. @prietyc123 Please go ahead and open an issue with the fresh logs.

Yes, I have raised a new issue

openshift-ci-robot added kind/bug Categorizes issue or PR as related to a bug. flake Categorizes issue or PR as related to a flaky test. labels Feb 6, 2020

amitkrout changed the title ~~[FLAKE] make test fails on osx in travis CI~~ make test fails on osx in travis CI Feb 6, 2020

amitkrout removed the flake Categorizes issue or PR as related to a flaky test. label Feb 6, 2020

amitkrout mentioned this issue Feb 6, 2020

Adds the debug info command #2518

Merged

amitkrout changed the title ~~make test fails on osx in travis CI~~ [FLAKE]make test fails on osx in travis CI Feb 6, 2020

openshift-ci-robot added the area/testing Issues or PRs related to testing, Quality Assurance or Quality Engineering label Feb 13, 2020

amitkrout added this to For consideration in Sprint 180 via automation Feb 18, 2020

openshift-ci-robot added the priority/High Important issue; should be worked on before any other issues (except priority/Critical issue(s)). label Feb 20, 2020

amitkrout mentioned this issue Feb 25, 2020

Updating nodejs:8 to nodejs:latest for source example run on openshift 4.* #2633

Merged

prietyc123 changed the title ~~[FLAKE]make test fails on osx in travis CI~~ [FLAKE]make test fails on osx and windows in travis CI Mar 4, 2020

prietyc123 mentioned this issue Mar 4, 2020

Unit test on windows #2562

Merged

prietyc123 assigned mik-dass Mar 4, 2020

openshift-ci-robot added the flake Categorizes issue or PR as related to a flaky test. label Mar 4, 2020

mik-dass mentioned this issue Mar 9, 2020

Fixes watch unit tests for windows #2692

Merged

mik-dass moved this from For consideration to In progress in Sprint 180 Mar 9, 2020

girishramnani added this to For consideration in Sprint 181 via automation Mar 11, 2020

girishramnani removed this from In progress in Sprint 180 Mar 11, 2020

girishramnani moved this from For consideration to In progress in Sprint 181 Mar 11, 2020

openshift-merge-robot closed this as completed in #2692 Mar 16, 2020

Sprint 181 automation moved this from In progress to Done Mar 16, 2020

mik-dass reopened this Mar 16, 2020

kadel moved this from Done to In progress in Sprint 181 Mar 16, 2020

kadel closed this as completed Apr 15, 2020

Sprint 182 automation moved this from In progress to Done Apr 15, 2020

kadel reopened this Apr 15, 2020

kadel moved this from Done to In progress in Sprint 182 Apr 15, 2020

mik-dass mentioned this issue Apr 16, 2020

Fixes watch test #2895

Merged

amitkrout closed this as completed in #2895 Apr 17, 2020

Sprint 182 automation moved this from In progress to Done Apr 17, 2020

prietyc123 reopened this May 20, 2020

prietyc123 added this to For consideration in Sprint 184 via automation May 20, 2020

prietyc123 added the needs more information label May 20, 2020

amitkrout closed this as completed May 21, 2020

Sprint 184 automation moved this from For consideration to Done May 21, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FLAKE]make test fails on osx and windows in travis CI #2570

[FLAKE]make test fails on osx and windows in travis CI #2570

prietyc123 commented Feb 6, 2020 •

edited

amitkrout commented Feb 6, 2020

mik-dass commented Feb 6, 2020 •

edited

kadel commented Feb 13, 2020

kadel commented Feb 20, 2020

prietyc123 commented Mar 6, 2020 •

edited

mik-dass commented Mar 16, 2020

amitkrout commented Mar 29, 2020

mik-dass commented Apr 13, 2020 •

edited

kadel commented Apr 14, 2020

kadel commented Apr 14, 2020

kadel commented Apr 15, 2020

mik-dass commented Apr 15, 2020

prietyc123 commented May 20, 2020

mik-dass commented May 20, 2020 •

edited

prietyc123 commented May 20, 2020

amitkrout commented May 20, 2020

mik-dass commented May 20, 2020 •

edited

amitkrout commented May 20, 2020

amitkrout commented May 20, 2020

mik-dass commented May 20, 2020

amitkrout commented May 20, 2020 •

edited

mik-dass commented May 20, 2020

mik-dass commented May 20, 2020 •

edited

amitkrout commented May 21, 2020

prietyc123 commented May 21, 2020

[FLAKE]make test fails on osx and windows in travis CI #2570

[FLAKE]make test fails on osx and windows in travis CI #2570

Comments

prietyc123 commented Feb 6, 2020 • edited

What versions of software are you using?

How did you run odo exactly?

Actual behavior

Expected behavior

Any logs, error output, etc?

amitkrout commented Feb 6, 2020

mik-dass commented Feb 6, 2020 • edited

kadel commented Feb 13, 2020

kadel commented Feb 20, 2020

prietyc123 commented Mar 6, 2020 • edited

mik-dass commented Mar 16, 2020

amitkrout commented Mar 29, 2020

mik-dass commented Apr 13, 2020 • edited

kadel commented Apr 14, 2020

kadel commented Apr 14, 2020

kadel commented Apr 15, 2020

mik-dass commented Apr 15, 2020

prietyc123 commented May 20, 2020

mik-dass commented May 20, 2020 • edited

prietyc123 commented May 20, 2020

amitkrout commented May 20, 2020

mik-dass commented May 20, 2020 • edited

amitkrout commented May 20, 2020

amitkrout commented May 20, 2020

mik-dass commented May 20, 2020

amitkrout commented May 20, 2020 • edited

mik-dass commented May 20, 2020

mik-dass commented May 20, 2020 • edited

amitkrout commented May 21, 2020

prietyc123 commented May 21, 2020

prietyc123 commented Feb 6, 2020 •

edited

mik-dass commented Feb 6, 2020 •

edited

prietyc123 commented Mar 6, 2020 •

edited

mik-dass commented Apr 13, 2020 •

edited

mik-dass commented May 20, 2020 •

edited

mik-dass commented May 20, 2020 •

edited

amitkrout commented May 20, 2020 •

edited

mik-dass commented May 20, 2020 •

edited