Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow defining field names that don't follow the restrictions of python variables #3266

Closed
dyeray opened this issue May 18, 2018 · 6 comments · Fixed by #3696
Closed

Allow defining field names that don't follow the restrictions of python variables #3266

dyeray opened this issue May 18, 2018 · 6 comments · Fixed by #3696

Comments

@dyeray
Copy link

dyeray commented May 18, 2018

Sometimes we may require defining field names that are not valid Python variables (for example containing spaces). I believe the correct way of doing this right now is that the developer needs to add a Pipeline that maps the internal name to the name we wan't to export. Scrapy could ship with a Pipeline (maybe enabled by default) that does this mapping for fields that define a field parameter. For example:

class MyItem(Item):
   date_from = Field(export_name='Initial date')

This is a very simple pipeline. However, since this is a very common use case, I think it is something that could be shipped by default. I'm happy to provide a PR for this.

@kmike
Copy link
Member

kmike commented Jun 1, 2018

If you're using dicts instead of scrapy.Item instances, you can do yield {'Initial date': my_date_from, ...}.

But I like the idea, I think we can it built-in.

@VMRuiz
Copy link
Contributor

VMRuiz commented Jul 20, 2018

I like that idea, but, wouldn't it be possible to reference the Item or the Item Loader by both the export_name or the var name instead of adding a new pipeline?

For example:

class MyItem(Item):
   date_from = Field(export_name='Initial date')

item = MyItem()
item['date_from'] = '01/01/1970' # This Works
item['Initial date'] = '01/01/1970' # This should work

loader = ItemLoader(item=MyItem()
loader.add_value('Initial date', '01/01/1970')  # This should work

@kmike
Copy link
Member

kmike commented Jul 4, 2019

#3696 also should solve this, though in a slightly different way.

@YngProgrammer
Copy link

Is there any PR to provide this by default as for now?
It could definitely improve the experience for many of us, rather than creating faulty pipelines and getting our hands dirty for no reason.

The "export_name" suggestion from @dyeray sounds like the perfect deal.

@kmike
Copy link
Member

kmike commented Jun 29, 2022

Regarding export_name idea, it's not solved by #3696. I wonder if we could make it a feature of https://github.com/scrapy/itemadapter.

@Gallaecio
Copy link
Member

itemadapter already supports field metadata. So we may only need to modify Scrapy to check the metadata, and if there is an export_name key, use it.

Shall we create a new issue about it, and maybe target 2.7?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants