MuleSoft WebCrawler Connector

MuleSoft AI Chain (MAC) Project

MAC Web Crawler

MuleSoft WebCrawler provides web crawling capabilities to extract data from web pages subsequently based on the structure of the website.

Requirements

• The maximum supported version for Java SDK is JDK 17.
• You can use JDK 17 only for running your application.
• Compilation with Java SDK must be done with JDK 11.

Configuration and Deployment

Installation (using maven central dependency)

<dependency>
   <groupId>io.github.mulesoft-ai-chain-project</groupId>
   <artifactId>mule4-webcrawler-connector</artifactId>
   <version>{version}</version>
   <classifier>mule-plugin</classifier>
</dependency>

Installation (building locally)

To use this connector, first build and install the connector into your local maven repository. Then add the following dependency to your application's pom.xml:

<dependency>
    <groupId>com.mulesoft.connectors</groupId>
    <artifactId>mule4-webcrawler-connector</artifactId>
    <version>{version}</version>
    <classifier>mule-plugin</classifier>
</dependency>

Installation into private Anypoint Exchange

You can also make this connector available as an asset in your Anypoint Exchange.

This process will require you to build the connector as above, but additionally you will need to make some changes to the pom.xml. For this reason, we recommend you fork the repository.

Then, follow the MuleSoft documentation to modify and publish the asset.

Deploying to CloudHub

In order for dynamic content retrieval to work in CloudHub based deployments, you will need to set the cloudhub.deployment property to true.

This can be done either via an application property in Runtime Manager, or in your CloudHub deployment configuration in your pom.xml.

This property will allow the installation of Chrome at runtime into your CloudHub 1.0 worker VM, or your CloudHub 2.0 container, along with the necessary dependencies.

Contributors ✨

Thanks goes to these wonderful people (emoji key):

Tommaso Bolis 💻 ⚠️ 👀 📦 📖	Shumon Sharif 💻 ⚠️ 👀 📦 📖	Amir Khan 💻 ⚠️ 👀 📦 📖	Yogesh Mudaliar 💻
Ryan Hoegg 💻 📦	Mihael Bosnjak 💻	saicharan0610 💻

This project follows the all-contributors specification. Contributions of any kind welcome!