-
Notifications
You must be signed in to change notification settings - Fork 96
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Spidermon fails to add _validation
field to the item.
#379
Comments
This is a limitation on the Item definition as it don't support adding new fields. In theory, using Alternatively, adding the field On my experience I had to just convert from Attrs class to |
That class is just an interface to simplify read and write access to the different types of Item classes. It doesn't add any additional functionality. I don't know if we could overcharge it to support additional fields not defined on the original item class. |
Maybe we can start by capturing the exception and raising a warning instead. |
I think this is the way to go. You either disable the creation of this |
@Gallaecio Shall we close this ticket as wont fix or similar? |
Well, I think there might be an action here even if we go with #379 (comment).
|
This can be turn off with We may expand the docs to explicitly state that if enabled, it will only work on items that have defined the Edit: Additionally, |
### Descrição Resolve #929 ### Comentários Atualizando minha venv local com os novos requirements, a execução de Prado-BA - mesmo raspador apontado na issue, coletando apenas desde agosto/23 até hoje, gera os seguintes arquivos: [prado_com_erro.csv](https://github.com/okfn-brasil/querido-diario/files/12612259/prado_com_erro.csv) - vazio [prado_com_erro.txt](https://github.com/okfn-brasil/querido-diario/files/12612261/prado_com_erro.txt) Ao fazer as atualizações propostas neste PR, temos (para o mesmo período de coleta): [prado_modificado.csv](https://github.com/okfn-brasil/querido-diario/files/12612271/prado_modificado.csv) - coleta com sucesso [prado_modificado.txt](https://github.com/okfn-brasil/querido-diario/files/12612273/prado_modificado.txt) Atualizar a biblioteca deprecada `pkg_resources` foi um dos erros apontados em #929. Segui as instruções em https://importlib-resources.readthedocs.io/en/latest/migration.html para isso. Pareceu funcionar. Quanto ao erro de validação apontado na issue #929, há uma issue aberta no repositório do Spidermon, que dialoga com o problema: scrapinghub/spidermon#379. Não tive certeza o que fazer com isso. Abro o PR como ponto de partida do que já enderecei, porém talvez haja outras verificações e modificações para fazer.
This problem was added by #358 and it is a backward incompatible change that we didn't inform our users that it was introduced. So we need to do something about it. Probably one new release that indicates that |
We already have this situation in Scrapy related to |
Ubuntu 22.04, Python 3.10.9, Spidermon 1.17.1
If an item defined with
attrs
library and this condition https://github.com/scrapinghub/spidermon/blob/master/spidermon/contrib/scrapy/pipelines.py#L140 isTrue
, then it fails here https://github.com/scrapinghub/spidermon/blob/master/spidermon/contrib/scrapy/pipelines.py#L141 with this message:The text was updated successfully, but these errors were encountered: