Welcome to searchmysite.net Discussions!

[m-i-l]

Welcome!

As per #6 hoping to get a bit of a discussion going about the future for searchmysite.net:

• Ask questions you’re wondering about.
• Share ideas.
• Engage with other community members.
• Welcome others and are open-minded. Remember that this is a community we
  build together.

[alcinnz]

Hi! I just wanted to say I've long been interested in the question of websearch, and I love SearchMySite.Net's vision!

[m-i-l]

Adrian, Thanks for your response. Internet search is interesting, and difficult. A lot of people think it is too big a problem for an unfunded startup working out of a garage (so to speak) now - while there were only just over 2 million websites when Google was working out of an actual garage in 1998, there are nearly 2 billion now. But I reckon it might be possible if you (i) don't even think about indexing it all because there's so much garbage now, and (ii) start small and aim to grow organically, e.g. with a niche search. Anyway, any bugs, ideas for enhancements, etc. gratefully accepted.

[ScootRay]

Michael, in a fit of good timing, stumbled upon your search engine and am really piqued by what I see so far. I've long been hunting around for an open source engine that was more focused on "curated" indexing and similar philosophies like those you wrote about in the blog portion of it.

I'm very much looking forward to watching this grow and I enjoyed your blog giving us an inside peek on how it's all unfolding. Am not a super-great-techie so I can only offer moral support. I'm also watching your expense reports, do be open when you want donations so we can pitch in.

I would suggest providing a link to the discussion forum from the search engine front page for more visibility and input from others.

Congrats on a very impressive job done so far!

Ray

[m-i-l]

Many thanks for your kind words and offer of support.

I like your suggestion of making the Discussion more visible, so I've logged #36 for this. It is a simple change so I'm hoping I'll be able to implement in the next few days.

[ScootRay]

Saw your recent post about removing Wikipedia. Great idea. I was going to suggest that and you beat me to it.

Thanks for all you do!
Ray

[m-i-l]

Thanks! It was mainly a cost saving thing to be honest (indexing an additional 16 million documents did require a fair bit of additional disk space), but the fact that it wasn't a universally lauded feature did help make it an easier decision.

[sbw]

I'm using Search My Site to drive my site search (https://sbw.org/search/). On that page is also >20 years of history on all the site search services I've used that eventually went away. I've brainstormed how I might build my own super simple site search, but so far I've never taken the time. So I'm glad to find a good service.

I look forward to seeing Search My Site as it grows and changes!

[m-i-l]

Interesting. I have to admit that I hadn't come across OpenSearchServer before. Sounds quite good in theory, but not hugely active.

I like Solr, and the way I use it here (docker with an XML config) you needn't know it is Java. Unfortunately it isn't that easy to get started with though, e.g. there's no clear guidance on basic questions like whether you should choose schema or schemaless.

[sbw]

And I wasn't aware of lunr.js. I just read a little about it, and it sounds very interesting! I like the idea of storing the index server-side as a static JSON file. That's similar to the static sqlite3 db I use for my Release 1.0 PDFs search page. Index once, use indefinitely!

But if I understand, lunr.js doesn't include a web crawler (just as Solr does not). The web crawler is a challenging part of a search service, I'm sure. I'm curious to read Search My Site's source code to learn more about its crawler.

[m-i-l]

I think people tend to use a GitLab CI pipeline or GitHub Action to build and deploy their lunr.js pagesData.json and searchIndex.json files on commit, similar to the way they'll have a CI pipeline to build and deploy their new Hugo or Jekyll or whatever static site on commit. Which is fine for static small sites, but not great for large static sites or dynamic sites.

searchmysite.net uses Scrapy for the crawler. There's a little bit about it at https://blog.searchmysite.net/posts/searchmysite.net-building-a-simple-search-for-non-commercial-websites/#indexing-to-solr-with-scrapy .

[sbw]

Thanks for the pointer to scrapy. I hadn't heard of it, and the crawler was the primary impediment to my building simple site search using sqlite or PostgreSQL.

On the blog, you ask for suggestions for domain names. I'll just say that lately I'm a fan of .ms, which had lots of short domains available last time I checked. I scored sbw.ms when the .org scandal threatened my longtime primary domain, sbw.org. I wrote a few comments about .ms on my page about the scandal.

[sbw]

searchmysite.net uses Scrapy for the crawler.

I'm a scrapy newbie, but I was interested to read about scrapy's built-in downloader HttpCacheMiddleware and its RFC 2616 Policy. Looking at the source code for that policy, it appears it'll send If-Modified-Since, which would allow my site to return HTTP 304 for most pages.

Of course, Search My Site shouldn't actually cache whole web pages, but I wonder whether a simple scrapy custom storage backend could use just the Last-Modified value from Solr to avoid downloading and reindexing pages that return 304.

That might speed up the crawl a lot. In my logs, I see I've returned 200 each time Search My Site has requested one of my pages that hasn't been updated lately.

Hope that's helpful!