Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[+1 MRG]look in ~/.config/scrapy.cfg for user config #1098

Merged
merged 1 commit into from Mar 26, 2015

Conversation

@nyov
Copy link
Contributor

@nyov nyov commented Mar 24, 2015

Just a slight enhancement.
I like to keep my ~ cleaner by having config files in ~/.config/.

@nramirezuy
Copy link
Contributor

@nramirezuy nramirezuy commented Mar 24, 2015

@nyov making a link from the project doesn't work ? or we can use an environment variable.

@nyov
Copy link
Contributor Author

@nyov nyov commented Mar 24, 2015

~/.config/ is a relatively standard path, like ~/.local/. It usually exists.
If I made a link (I assume you mean a symlink), then that would be counter to having a cleaner ~/ - there'd still be a file there.

@nramirezuy
Copy link
Contributor

@nramirezuy nramirezuy commented Mar 24, 2015

I meant a symlink inside the project not on the home. But I didn't know it was standard, do you have some source?

Either way I would like to see an implementation using a environment variable.

@nyov
Copy link
Contributor Author

@nyov nyov commented Mar 24, 2015

Here's the standard: http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html#variables
These locations conform with $XDG_DATA_HOME and $XDG_CONFIG_HOME.

If $XDG_CONFIG_HOME is either not set or empty, a default equal to $HOME/.config should be used.

So according to this spec, I should check the $XDG_CONFIG_HOME env var, and only use the fixed ~/.config/scrapy.cfg-path if it's not set.
I'll add this in a bit.

@nyov nyov force-pushed the nyov:nyov/userconfig branch from e26386b to deb6124 Mar 24, 2015
@@ -5,6 +5,11 @@
import six
from six.moves.configparser import SafeConfigParser

try:
from xdg.BaseDirectory import xdg_config_home

This comment has been minimized.

@kmike

kmike Mar 24, 2015
Member

what's xdg?

@nyov
Copy link
Contributor Author

@nyov nyov commented Mar 25, 2015

xdg is "pyxdg" http://freedesktop.org/wiki/Software/pyxdg/
or known as "python-xdg" in debian/ubuntu: https://packages.debian.org/jessie/python-xdg

@kmike
Copy link
Member

@kmike kmike commented Mar 25, 2015

@nyov I think we can't use pyxdg because it is GPL and doesn't support Python 3.

@nyov
Copy link
Contributor Author

@nyov nyov commented Mar 25, 2015

Should I rather not depend on that package? The functionality is defined as this and not very likely to change:

_home = os.path.expanduser('~')
xdg_config_home = os.environ.get('XDG_CONFIG_HOME') or os.path.join(_home, '.config')

Oh you're right. License issues, heh. I'll change it.

@nyov nyov force-pushed the nyov:nyov/userconfig branch from deb6124 to e852337 Mar 25, 2015
@nyov
Copy link
Contributor Author

@nyov nyov commented Mar 25, 2015

Would this work now?

@nyov nyov force-pushed the nyov:nyov/userconfig branch from e852337 to 0cef9f0 Mar 25, 2015
@kmike
Copy link
Member

@kmike kmike commented Mar 25, 2015

LGTM. I personally don't care about this feature, but see zero reasons not to merge it. If it makes life easier for people why not do it.

By the way, this scrapy.cfg lookup chain is not documented anywhere :) Docs only say about scrapy.cfg in the project root.

@nyov
Copy link
Contributor Author

@nyov nyov commented Mar 25, 2015

I would add some docs for it, but I can't find a good category?

@kmike
Copy link
Member

@kmike kmike commented Mar 25, 2015

There are some docs for scrapy.cfg in commands.rst ("Default structure of Scrapy projects"); adding a note there would be nice.

@nyov nyov force-pushed the nyov:nyov/userconfig branch from 0cef9f0 to b1c0aad Mar 26, 2015
@nyov
Copy link
Contributor Author

@nyov nyov commented Mar 26, 2015

Alright, I decided to write it under a new heading, hope that works.

@nramirezuy nramirezuy changed the title look in ~/.config/scrapy.cfg for user config [+1 MRG]look in ~/.config/scrapy.cfg for user config Mar 26, 2015
@kmike
Copy link
Member

@kmike kmike commented Mar 26, 2015

Thanks @nyov!

  • What about mentioning that the settings from all available scrapy.cfg files are merged?
  • Regarding env variables - "Scrapy also understands, and can be configured through, a number of environment variables." doesn't help users because it is not documented which variables Scrapy understands. I see the code which allows to override settings with SCRAPY_... env variables is marked for deprecation/removal. The only relevant env variables seems to be SCRAPY_SETTINGS_MODULE and SCRAPY_PROJECT. It could be better to document them explicitly.

@nyov @nramirezuy what do you think? I can merge the changes and tweak the docs if this sounds ok (updated PR is also welcome, of course :).

@nramirezuy
Copy link
Contributor

@nramirezuy nramirezuy commented Mar 26, 2015

👍

@nyov
Copy link
Contributor Author

@nyov nyov commented Mar 26, 2015

@kmike, I was afraid of that when I rwrote the sentence ;) Okay I will list the ENV vars you mentioned, as I'm not sure what all SCRAPY_... would include.

@nyov nyov force-pushed the nyov:nyov/userconfig branch from b1c0aad to 1134a9c Mar 26, 2015
@kmike
Copy link
Member

@kmike kmike commented Mar 26, 2015

Thanks!

kmike added a commit that referenced this pull request Mar 26, 2015
[+1 MRG]look in ~/.config/scrapy.cfg for user config
@kmike kmike merged commit 39085ae into scrapy:master Mar 26, 2015
1 check passed
1 check passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@nyov nyov deleted the nyov:nyov/userconfig branch Mar 27, 2015
@nyov nyov mentioned this pull request Apr 27, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants
You can’t perform that action at this time.