Adding Knowledge Center Libraries (Kiwix Dev Setup)

Adding Knowledge Center Libraries (Kiwix Dev Setup)

This page provides a step-by-step guide for configuring your local dev machine to point to the production Kiwix instance and downloading/adding new Knowledge Center libraries.

---

Table of Contents

• Purpose
• Steps Overview
• Step 1: Update the Kiwix Library Limit
• Step 2: Update the Docker Compose URL
• Step 3: Update the Proxy Handler
• Step 4: Update the Database URL
• Step 5: Start Your Containers
• Step 6: Trigger the Kiwix Scrape Job

---

Purpose

• Point Local Dev at Production Kiwix: Configure your local environment to use the production Kiwix instance so that real library content is available.
• Add New Libraries: Trigger the scrape job to pull down and register new Knowledge Center libraries.

---

Steps Overview

1. Increase the Kiwix library fetch limit in code.
2. Update the Kiwix URL in docker-compose.yml.
3. Update the proxy handler to use HTTPS.
4. Update the database URL for the Kiwix open content provider.
5. Start your containers.
6. Use the nats CLI to trigger the scrape job.

---

Step 1: Update the Kiwix Library Limit

Open kiwix.go and navigate to line 23. Change the return value to allow fetching a larger library set:

return 1000

This increases the limit on how many libraries are fetched from the Kiwix instance.

---

Step 2: Update the Docker Compose URL

Open docker-compose.yml and navigate to lines 90-91. Update the Kiwix URL and CORS URL to point to the dev/production instance:

KIWIX_SERVER_URL=https://kiwix.dev.unlockedlabs.xyz
CORS_ALLOWED_ORIGINS=http://localhost,http://127.0.0.1,https://kiwix.dev.unlockedlabs.xyz

---

Step 3: Update the Proxy Handler

Open proxy_handler.go and navigate to line 57. Change the scheme from http to https:

// Change:
http
// To:
https

This ensures proxy requests to Kiwix are made over a secure connection.

---

Step 4: Update the Database URL

Run the following query against the unlocked database to update the stored Kiwix URL for the open content provider:

UPDATE open_content_providers
SET url = 'https://kiwix.dev.unlockedlabs.xyz'
WHERE id = 2;

---

Step 5: Start Your Containers

Bring up your local environment:

make dev or make init 

Ensure all containers start successfully before proceeding.

---

Step 6: Trigger the Kiwix Scrape Job

You will need the nats CLI tool installed. If it is not already installed, search for installation instructions for your platform.

Also you will need to obtain the scrape_kiwix job ID which can be obtained by running the following SQL query:

select id from cron_jobs where name = 'scrape_kiwix';

Once nats is available, run the following command to publish the scrape job:

nats pub --server=nats://127.0.0.1:4222 --user=unlocked --password=dev tasks.scrape_kiwix '{"job_id":"@job_id_here@", "open_content_provider_id":2}'

If this command does not work, verify the correct server address, credentials, and open_content_provider_id for your environment before retrying.

Success: The scrape job will run and new Kiwix libraries will be pulled into your local Knowledge Center.