-
Notifications
You must be signed in to change notification settings - Fork 10.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: batch_path_differ
test
#5639
Conversation
Codecov Report
@@ Coverage Diff @@
## master #5639 +/- ##
==========================================
- Coverage 88.85% 88.83% -0.02%
==========================================
Files 162 162
Lines 10964 10964
Branches 1894 1894
==========================================
- Hits 9742 9740 -2
- Misses 942 943 +1
- Partials 280 281 +1
|
I also noticed this error being intermittent |
I am OK with merging this for now as a workaround, but I would not close #5633 with it. I think we still need to find out why this is happening, where the extra item comes from without the delay. |
@Gallaecio Is not a extra item that is yielded on Windows. Here is the output of
4 items, 1 is empty that the reason that test have the: scrapy/tests/test_feedexport.py Line 2308 in 07b079d
Since we only have 3 items declared: scrapy/tests/test_feedexport.py Lines 2294 to 2298 in 07b079d
On Windows this is the output we have for
As you can see we don't have the empty item in list on Windows. |
@Gallaecio I refactored the test, now should work cross all environments. |
batch_path_differ
test in Windowsbatch_path_differ
test
Do you know why are we getting an empty item on Linux? It's mystery to me :) |
@kmike The test generate 4 files, one of them is empty, this result in an empty item in list, since we parse all the files even the empty ones, we could add a validation in the
scrapy/tests/test_feedexport.py Lines 2033 to 2038 in 07b079d
I pushed other change doing the second thing that you suggested. |
To me the question is: why is the empty file not generated in Windows? Not generating an empty file when using batch exports would be a nice-to-have feature, but I was under the impression that we did not have that feature yet implemented, and the |
@kmike @Gallaecio I add sleep again because even we filtering out the empty item we still have diff in output on windows 3 != 2, because on Windows instead of creating 3 files it created 2 and output the last 2 items in the last file with an invalid syntax: |
Nice find @Laerte! I think it could make sense to make this test XFAIL on WIndows, and open a new issue for the problem you found. |
@kmike I used |
Congrats for making the test suite green @Laerte! :) What was the issue with xfail, how were you using it? |
This is how i tried: |
Aha, so the test is not always failing, and so when |
Indeed i tried without |
any additional updates needed before this PR ready to be merged? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good, thanks @Laerte!
@gliptak the PR looks good to me. Next step is to get a second approval from one of the Scrapy maintainers. |
Workaround for #5633
While debugging in Windows to identify the root cause i noticed that the test don't fail. So it seems that in Windows the test run too fast and this result in a invalid output, so in order to fix the test i add a delay only for Windows and this make the test suceed, since now the output is correct.Check #5639 (comment) for details.