Skip to content

Conversation

tmichaeldb
Copy link
Contributor

@tmichaeldb tmichaeldb commented Aug 15, 2024

This PR adds a new WebConfig subclass of DataSourceConfig. Specifically:

  • Updated agent.add_webpages to support crawl_depth and regex based URL filters
  • Added WebConfig class that configures web crawling a list of URLs for Minds

Part of ML-96

@tmichaeldb tmichaeldb self-assigned this Aug 15, 2024
@tmichaeldb tmichaeldb added the enhancement New feature or request label Aug 15, 2024
Copy link

Coverage

Coverage Report
FileStmtsMissCoverMissing
mindsdb_sdk
   agents.py2075275%33, 101, 104, 112, 120, 138, 156, 166, 169, 173, 175, 177, 179, 181, 183, 245, 258, 262–277, 289, 298–302, 309–310, 356, 368–369, 419, 468–469, 472, 479, 500–502, 506–510
   databases.py45296%109, 137
   handlers.py39197%77
   jobs.py97793%41, 53, 81, 85, 147–150
   knowledge_bases.py1261588%58–61, 155, 159–166, 171–175, 254–257, 326, 343
   ml_engines.py42393%94, 126, 128
   models.py2101991%109, 140–141, 222, 231, 233, 303, 339, 348, 372, 397, 491, 499, 518, 534, 542, 567, 571, 584
   projects.py64198%160
   query.py13192%14
   skills.py53394%43, 45, 49
   tables.py1301588%140–142, 145, 165, 192, 203–204, 209, 224, 227, 321, 342–347, 356, 376
   views.py37295%105, 138
mindsdb_sdk/connectors
   rest_api.py2254281%19–29, 35–36, 51, 54–55, 75–77, 98, 101, 108–111, 144–152, 173–174, 209–212, 226–227, 281–286
mindsdb_sdk/utils
   mind.py41410%1–119
   openai.py853065%37–40, 83–85, 107, 148–158, 215–216, 234–240, 258–276
   table_schema.py21210%1–54
TOTAL152225583% 

Tests Skipped Failures Errors Time
23 0 💤 0 ❌ 0 🔥 10.293s ⏱️

Copy link
Contributor

@dusvyat dusvyat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@tmichaeldb tmichaeldb merged commit fe6b165 into main Aug 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: merged
Development

Successfully merging this pull request may close these issues.

2 participants