db_load_utils.py do I need to add to INCLUDE_TYPES? or can it be left empty to get all?

[Telemarker77]

in code/tools/db_load_utils.py

what is the "INCLUDE TYPES" for if it simply gets all that are not listed in "SKIP_TYPES"

Item type categorization

SKIP_TYPES = ["ItemList", "ListItem", "AboutPage", "Person"]

INCLUDE_TYPES = [
"FAQ", "WebPage", "Question", "Answer", "Organization", "BlogPosting", "MedicalWebPage", "HealthTopicContent", "MedicalOrganization", "OverviewHealthAspect"

I've noticed that the scraper will pick up image objects event if not added to INCLUDE_TYPES.

Should I simply Leave INCLUDE_TYPES as an empty array?

In the source Repo you have:

Item type categorization

SKIP_TYPES = ["ItemList", "ListItem", "AboutPage", "WebPage", "WebSite", "Person"]

INCLUDE_TYPES = [
"Recipe", "NeurIPSPoster", "InvitedTalk", "Oral", "Movie", "LocalBusiness", "Review",
"TVShow", "TVEpisode", "Product", "Offer", "PodcastEpisode", "Book",
"Podcast", "TVSeries", "ProductGroup", "Event", "FoodEstablishment",
"Apartment", "House", "Home", "RealEstateListing", "SingleFamilyResidence", "Offer",
"AggregateOffer", "Event", "BusinessEvent", "Festival", "MusicEvent", "EducationEvent",
"SocialEvent", "SportsEvent"

I can't work out why you have added these Types if it gets all by default?

[chelseacarter29]

Hi @Telemarker77 - what is the command you are using? What are you trying to do more specifically?

[Telemarker77]

when I want to load any data in to any db I will either use:

python -m tools.db_load
or
python -m scraping.incrementalCrawlAndLoad example.com

when I do either of these commands is code/tools/db_load_utils.py called?

and does it matter if I leave both empty like

SKIP_TYPES = []

INCLUDE_TYPES = []

There are no Types I want to skip and when I have tried it seems to recognise schema.org types when I have left INCLUDE_TYPES as an empty array.

as the repo includes all sorest of Types I was confused as to whether I need to add the Types I want it to include or not.

what is the difference between leaving both arrays empty OR adding a whole bunch of Types like we see in the Microsoft/NLWeb repo

Thanks for your reply

Josh A

[chelseacarter29]

What type of a file are you trying to load? Just a webpage? If webpage or RSS feed, you don't need to specify anything regarding the types, but if you're trying to use something different I may need an example.

I think those may be there for some early scenarios/filtering one might want to do - but we don't generally use them when building demos.

[genericrobot77]

so if i was loading a test jsonl file using python -m tools.db_load can I assume that the load toll will pick out the "MedicalOrganization" Type, "Place" Type, "PostalAddress" Type, etc. when leaving the
SKIP_TYPES = []
INCLUDE_TYPES = []

as empty arrays?

I cannot attach jsonl files here so am copying the contents below.

{"@context":"https://schema.org","@type":"MedicalOrganization","location":{"@type":"Place","address":{"@type":"PostalAddress","streetAddress":"130 Avoca Street","addressLocality":"RANDWICK","addressRegion":"NSW","postalCode":"2031","addressCountry":"AUS"}},"medicalSpecialty":[{"@type":"MedicalSpecialty","name":"GP (General practice)"}],"additionalProperty":[{"@type":"PropertyValue","name":"Accessible Toilet","value":true},{"@type":"PropertyValue","name":"Toilet","value":true},{"@type":"PropertyValue","name":"Wheelchair Access","value":true},{"@type":"PropertyValue","name":"Fees Apply","value":true}],"name":"Avoca Street Medical Centre","@id":"98af54db7a30aaee76128db47aa9968ce89929dbc42fcb2f51a7db75d96e7bbc","ContactPoint":[{"@type":"ContactPoint","telephone":"02 9399 3335","contactType":"work"},{"@type":"ContactPoint","url":"www.asmc.net.au","contactType":"work"}]}
{"@context":"https://schema.org","@type":"Place","@id":"98af54db7a30aaee76128db47aa9968ce89929dbc42fcb2f51a7db75d96e7bbc","openingHoursSpecification":[{"@type":"OpeningHoursSpecification","dayOfWeek":"monday","opens":"08:00:00","closes":"17:00:00"},{"@type":"OpeningHoursSpecification","dayOfWeek":"tuesday","opens":"08:00:00","closes":"17:00:00"},{"@type":"OpeningHoursSpecification","dayOfWeek":"wednesday","opens":"08:00:00","closes":"17:00:00"},{"@type":"OpeningHoursSpecification","dayOfWeek":"thursday","opens":"08:00:00","closes":"17:00:00"},{"@type":"OpeningHoursSpecification","dayOfWeek":"friday","opens":"08:00:00","closes":"17:00:00"}]}