Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[MGR+1] Change, document LOG_UNSERIALIZABLE_REQUESTS #1610

Merged
merged 4 commits into from Jul 29, 2016
Merged

[MGR+1] Change, document LOG_UNSERIALIZABLE_REQUESTS #1610

merged 4 commits into from Jul 29, 2016

Conversation

darshanime
Copy link
Contributor

Changed LOG_UNSERIALIZABLE_REQUESTS to SCHEDULER_DEBUG. Document it in the docs. Changed behavior to log once and collect stats.
Fixes #1592

@codecov-io
Copy link

codecov-io commented Nov 21, 2015 •
edited

Current coverage is 83.44% (diff: 28.57%)

Merging #1610 into master will decrease coverage by 0.02%

Powered by Codecov. Last update ec1c615...d8e62e6

@darshanime darshanime changed the title Change, document LOG_UNSERIALIZABLE_REQUESTS [WIP] Change, document LOG_UNSERIALIZABLE_REQUESTS Nov 21, 2015
@darshanime
Copy link
Contributor Author

Kindly review @curita, @nramirezuy.
The codecov is complaining because I need to write tests yet, correct ?
Where should they go ? (I am thinking about test.util_reqser.py)

nramirezuy
nramirezuy reviewed Nov 23, 2015
View reviewed changes
scrapy/core/scheduler.py
@@ -28,7 +28,7 @@ def from_crawler(cls, crawler):
dupefilter = dupefilter_cls.from_settings(settings)
dqclass = load_object(settings['SCHEDULER_DISK_QUEUE'])
mqclass = load_object(settings['SCHEDULER_MEMORY_QUEUE'])
logunser = settings.getbool('LOG_UNSERIALIZABLE_REQUESTS')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We never completely remove older cases. This settings should still remain in the code with a deprecation warning.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made the change in the latest commit

@darshanime
Copy link
Contributor Author

@nramirezuy, can you please review the PR? Thanks.

@lopuhin
Copy link
Member

lopuhin commented May 23, 2016

Anything blocking the merge here?
I have no opinion on the best name for the setting, but at least mentioning the setting in the docs could save a lot of debugging time (as the serialization error could be raised in several different places).

redapple
redapple reviewed Jul 25, 2016
View reviewed changes
docs/topics/jobs.rst
@@ -96,4 +96,6 @@ But this will::
somearg = response.meta['somearg']
print "the argument passed is:", somearg

If you wish to log the requests that couldn't be serialized, you can set the ``SCHEDULER_DEBUG`` setting to ``True`` in the project's settings page. It is ``False`` by default.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you wrap the line around 80 chars?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay. Also, I'll pull in the latest code and merge the conflict. Is there anything else?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay. Also, I'll pull in the latest code and resolve the conflict. Is there anything else?

@darshanime
Copy link
Contributor Author

darshanime commented Jul 25, 2016 •
edited

@redapple kindly review!
Travis builds, codecov wants tests:/

redapple
redapple reviewed Jul 25, 2016
View reviewed changes
docs/topics/settings.rst
Default: ``False``

Setting to ``True`` will log debug information about the requests scheduler.
This currently logs(only once) if the requests cannot be serialized to disk.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

missing space: logs (only once)

@redapple
Copy link
Contributor

LGTM

@darshanime darshanime changed the title [WIP] Change, document LOG_UNSERIALIZABLE_REQUESTS [MGR] Change, document LOG_UNSERIALIZABLE_REQUESTS Jul 29, 2016
@darshanime
Copy link
Contributor Author

@redapple, can we merge this?

@redapple redapple changed the title [MGR] Change, document LOG_UNSERIALIZABLE_REQUESTS [MGR+1] Change, document LOG_UNSERIALIZABLE_REQUESTS Jul 29, 2016
@redapple
Copy link
Contributor

Fine by me. @kmike , @eliasdorneles , @dangra , what do you think?

@dangra
Copy link
Member

dangra commented Jul 29, 2016

LGTM.

@eliasdorneles
Copy link
Member

Looks good, merging -- thanks @darshanime !

@eliasdorneles eliasdorneles merged commit 34e7dad into scrapy:master Jul 29, 2016
@darshanime darshanime deleted the scheduler_debug branch July 29, 2016 15:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
7 participants
@darshanime @codecov-io @lopuhin @redapple @dangra @eliasdorneles @nramirezuy