Enable overriding TEMPLATE_DIR in startproject

[SudShekhar]

Fix for Issue #671.
Kindly let me know if there any changes are required.

Thanks!

[nramirezuy]

You can use self.setings.get('TEMPLATES_DIR', join(scrapy.__path__[0], 'templates')).

I feel like the default should be present on default_settings.py

[nramirezuy]

I been doing some investigation on this.

1. We want to create a new setting on default_settings.py right before TEMPLATES_DIR called TEMPLATES_DIR_BASE and with a value of os.path.join(scrapy.__path__[0], 'templates').
2. Let's introduce 4 new settings one per template type TEMPLATES_PROJECT and TEMPLATES_SPIDERS and his respective *_BASE. The non-base ones doesn't need to be present on default_settings.py just on documentation.
3. Create a function on utils. Taking into account that:

• *_BASE settings are our defaults and should always be present. (settings['X_BASE'] is allowed)
• Specific templates can be absolute or relative. But relative to current position (just call abspath), but if not exists we should try by joining with TEMPLATES_DIR and after TEMPLATES_DIR_BASE.

This might be like a little confusing, so if you have doubts or comments don't hesitate to contact me.

[SudShekhar]

I have made changes to the code. Please have a look.

Also, I am not sure where TEMPLATES_DIR_BASE will be used, since we aren't removing TEMPLATES_DIR from the default configuration (or are we?). This is why I didn't add the documentation for TEMPLATES_DIR_BASE. I will add it once we review the other things.

[nramirezuy]

Respect the spaces = 😄

[nramirezuy]

I just added this to Scrapy 1.0 so we can break backwards compatibility.

[nramirezuy]

The function should look something like this:

from scrapy.utils.misc import args_to_iter

def get_template_dir(settings, name='project'):
    if name == "project":
        folder_name = settings.get('TEMPLATES_PROJECT', os.path.join(base_dir, settings['TEMPLATES_PROJECT_BASE']))
    else:
        folder_name = settings.get('TEMPLATES_SPIDERS', os.path.join(base_dir, settings['TEMPLATES_SPIDERS_BASE']))
     if os.path.isabs(folder_name):
         return folder_name
     else:
         for base_dir in args_to_iter(settings.get('TEMPLATES_DIR', settings['TEMPLATES_DIR_BASE'])):
             path = os.path.join(base_dir, folder_name)
             if os.path.exists(path):
                 return path

[nramirezuy]

If you can make some tests for our new function, that would be great.

[nramirezuy]

remove get

[nramirezuy]

How do you feel about supporting multiple folders? Our function returning existent folders and the command being able to look into these folders?

/cc @SudShekhar @kmike

[nramirezuy]

I noticed that both of this path are the same. We need to remove one, I purpose the second one since it requires an extra import.
After one is removed, we have to add a path to the project in the position 0. I already found a way of getting it, but it needs more changes. So I will let you do some research, maybe you can find a better option. 😄

[SudShekhar]

I looked into ways to do this. My idea would be to use os.getcwd() to get the directory where the script is running and add this to the user's version of scrapy.cfg (append at the end or something). Is this what you had in mind?

[nramirezuy]

But doesn't work if you are calling scrapy command from inside the module.

[SudShekhar]

I was suggesting that when we create the project, we add this setting to the scrapy.cfg (using startproject.py). So when we are running this command from inside the module (for genspider), you will automatically get the folder location as a part of template_dir_base right? I guess I am a little confused by what you mean by module.

[SudShekhar]

@nramirezuy : What if we keep only the first line in TEMPLATE_DIR_BASE and we modify the get_template_dir function, to look through three stages:

1. TEMPLATES_PROJECT -- template folder input given by user
2. For startproject, we look at abspath(join(dirname(__file__), '..', 'templates')) [folder where we run the command]. For genspider, we find closest scrapy.cfg and use its directory.
3. TEMPLATES_PROJECT_BASE -> the original scrapy template folder.

Please let me know what do you think.

[nramirezuy]

@SudShekhar Since we are deprecating the _BASE settings in favor of using priorities on Settings, I think we should do the same. But still keep the setting as list.

[redapple]

Closing in favor of #1575