Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The crawler shouldn't create YAML files #243

Closed
bfabio opened this issue Aug 30, 2021 · 1 comment
Closed

The crawler shouldn't create YAML files #243

bfabio opened this issue Aug 30, 2021 · 1 comment
Assignees
@bfabio

Comments

@bfabio
Copy link
Member

bfabio commented Aug 30, 2021

The crawler should not create YAML files
(softwares.yml, amministrazioni.yml, software_categories.yml, software-open-source.yml, software-riuso.yml, software_scopes.yml, software_tags.yml), instead the site should query directly the API and build the data it needs.

Pros:

  • Single source of truth
  • Decouples the crawler from the website
  • Ability to switch the API server in a preview build of the website
bfabio added a commit to bfabio/developers.italia.it that referenced this issue May 11, 2022
@bfabio
refactor: don't get the list of possible scopes from the crawler
1d4766a 
Don't get the list of possible publiccode's scopes from the crawler,
it's a static list defined in the Standard (https://yml.publiccode.tools).

Related to italia/publiccode-crawler#243
@bfabio bfabio mentioned this issue May 11, 2022
Merged
bfabio added a commit to bfabio/developers.italia.it that referenced this issue May 11, 2022
@bfabio
refactor: use publiccode from local data file
48be271 
Related to italia/publiccode-crawler#243.
This was referenced May 11, 2022
Merged
Merged
@bfabio
Copy link
Member Author

bfabio commented May 11, 2022

software_tags.yml is not created any longer, it's just on crawler.developers.italia.it from an old run. We can remove it once italia/developers.italia.it#1200 is in.

bfabio added a commit to bfabio/publiccode-crawler that referenced this issue May 11, 2022
@bfabio
refactor: don't create software_scopes.yml and software_categories.yml
34ceb90 
Related to italia#243
@bfabio bfabio mentioned this issue May 11, 2022
Merged
bfabio added a commit that referenced this issue May 11, 2022
@bfabio
refactor: don't create software_scopes.yml and software_categories.yml (
502d320 
#270)

Related to #243
bfabio added a commit to bfabio/developers.italia.it that referenced this issue May 11, 2022
@bfabio
refactor: use publiccode from local data file
b17b0c7 
Related to italia/publiccode-crawler#243.
bfabio added a commit to bfabio/developers.italia.it that referenced this issue May 11, 2022
@bfabio
refactor: don't get the list of possible scopes from the crawler
46ecba2 
Don't get the list of possible publiccode's scopes from the crawler,
it's a static list defined in the Standard (https://yml.publiccode.tools).

Related to italia/publiccode-crawler#243
bfabio added a commit to italia/developers.italia.it that referenced this issue May 12, 2022
@bfabio
refactor: don't get the list of possible scopes from the crawler (#1201)
192492c 
Don't get the list of possible publiccode's scopes from the crawler,
it's a static list defined in the Standard (https://yml.publiccode.tools).

Related to italia/publiccode-crawler#243
@bfabio bfabio mentioned this issue May 28, 2022
Closed
This was referenced Jul 30, 2022
Closed
Open
bfabio added a commit to bfabio/publiccode-crawler that referenced this issue Aug 31, 2022
@bfabio
refactor!: remove the YML exports and elasticsearch dependency
7bfb4fe 
Fix italia#243.
bfabio added a commit to bfabio/publiccode-crawler that referenced this issue Aug 31, 2022
@bfabio
refactor!: remove the YML exports and elasticsearch dependency
674c9fc 
Fix italia#243.
bfabio added a commit to bfabio/publiccode-crawler that referenced this issue Sep 12, 2022
@bfabio
refactor!: remove the YML exports and elasticsearch dependency
77cb48b 
Fix italia#243.
bfabio added a commit to bfabio/publiccode-crawler that referenced this issue Sep 14, 2022
@bfabio
refactor!: remove the YML exports and elasticsearch dependency
525e01f 
Fix italia#243.
bfabio added a commit to bfabio/publiccode-crawler that referenced this issue Oct 6, 2022
@bfabio
refactor!: remove the YML exports and elasticsearch dependency
9fe80c1 
Fix italia#243.
@bfabio bfabio self-assigned this Oct 13, 2022
bfabio added a commit to bfabio/publiccode-crawler that referenced this issue Oct 13, 2022
@bfabio
refactor!: remove the YML exports and elasticsearch dependency
7a47fd8 
Fix italia#243.
bfabio added a commit to bfabio/publiccode-crawler that referenced this issue Oct 13, 2022
@bfabio
refactor!: remove the YML exports and elasticsearch dependency
97d201f 
Fix italia#243, Fix italia#291.
bfabio added a commit to bfabio/publiccode-crawler that referenced this issue Oct 13, 2022
@bfabio
refactor!: remove the YML exports and elasticsearch dependency
1e3d167 
Fix italia#243, Fix italia#291.
@bfabio bfabio closed this as completed in 1c8fd9d Oct 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bfabio bfabio

Labels
None yet
Projects
Developers Italia-next
Status: ✅ Done
Milestone
No milestone
Development

No branches or pull requests
1 participant
@bfabio