Distributed Peer-to-Peer Web Search Engine and Intranet Search Appliance
Java HTML JavaScript CSS XSLT Shell Other
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
.settings migrated Solr 5.5 -> Solr 6.6 and from Java 1.7 -> 1.8 Jun 9, 2017
addon Merge pull request #98 from Velociraptor85/patch-2 Jan 26, 2017
bin Extended Mediawiki dump import to remote URLs. Apr 14, 2017
debian Updated Debian optional dependencies with the ones used for snapshots Oct 5, 2017
defaults update bootstrap hosts Jul 1, 2018
dictionaries changed way to integrate dictionary files: Sep 2, 2009
docker Removed Docker Cloud deploy button as service will soon be shut down Apr 18, 2018
examples move example code SearchClient out of yacycore package Mar 14, 2016
htroot Fixed rendering of crawl queues page for URLs with raw IPV6 addresses Aug 13, 2018
langdetect Use language-detection library for increased accuracy Jul 2, 2015
lib Upgraded Apache PDFBox dependency from 2.0.9 to 2.0.11 Aug 18, 2018
libbuild Use standard Java annotation syntax instead of custom Javadoc tag Jul 13, 2018
libt update to JUnit 4.11 Jul 6, 2014
locales Added a link to MediaWiki dumps summary in import page for convenience Aug 8, 2018
skins correct the dark themes to show also a dark navbar on searchresults Jun 4, 2015
source Updated pdf cache clear steps consistently with current pdfbox version Aug 16, 2018
test Added a parser for XZ compressed archives. Aug 15, 2018
vocabularies moved yacy.logging to defaults according to request in May 17, 2012
.checkstyle problems with code style Jun 29, 2009
.classpath Upgraded Apache PDFBox dependency from 2.0.9 to 2.0.11 Aug 18, 2018
.env Removed unnecessary variables for local heroku run Jul 12, 2016
.gitignore Ignore generated Javadoc with git SCM. Jan 9, 2017
.project Update .project May 8, 2016
.travis.yml Fixed Travis configuration for Debian package building task Oct 6, 2017
AUTHORS added flori and me Jun 13, 2008
COPYRIGHT updated copyright message; included LGPL for 'cora' and a warranty Jun 30, 2013
Heroku.md Simplified Heroku variables configuration Jul 28, 2016
NOTICE * updated jxpath to latest v1.3 Jul 15, 2009
Procfile Simplified Heroku variables configuration Jul 28, 2016
README.md Removed Docker Cloud deploy button as service will soon be shut down Apr 18, 2018
app.json Updated YaCy home page embedded links from http to https scheme May 22, 2018
assembly.xml Fixed maven assembly base directory to match last main YaCy binaries. Jan 9, 2017
build.nsi Updated the JRE URL from 8u171 to 8u181 for the MS Windows installer Aug 14, 2018
build.properties migrated Solr 5.5 -> Solr 6.6 and from Java 1.7 -> 1.8 Jun 9, 2017
build.xml Upgraded Apache PDFBox dependency from 2.0.9 to 2.0.11 Aug 18, 2018
getWin32MaxHeap.bat update for memory observer algorithm Dec 6, 2009
gpl.txt initial load with yacy 0.36 Apr 7, 2005
installYaCyWindowsService.bat upd classpath in batches (remove not necessary htroot) Feb 3, 2016
killYACY.sh Fix for http://mantis.tokeek.de/view.php?id=432 Aug 25, 2014
lgpl21.txt migrated all my LGPL 3 -licensed files to the LGPL 2.1 because LGPL 3… Jun 28, 2010
pom.xml Upgraded Apache PDFBox dependency from 2.0.9 to 2.0.11 Aug 18, 2018
reconfigureYACY.sh Fix for http://mantis.tokeek.de/view.php?id=432 Aug 25, 2014
startYACY.bat upd classpath in batches (remove not necessary htroot) Feb 3, 2016
startYACY.sh Added a start script option to run as a foreground process without JMX Apr 17, 2018
startYACY_debug.bat harmonize classpath with startYaCy.bat Feb 28, 2016
stopYACY.bat upd classpath in batches (remove not necessary htroot) Feb 3, 2016
stopYACY.sh Updated log path in informative message of stop script. Nov 14, 2017
uninstallYaCyWindowsService.bat added Windows Service installer Dec 26, 2013
updateYACY.sh Fix for http://mantis.tokeek.de/view.php?id=432 Aug 25, 2014
yacy-packages.readme added documentation for new yacy package structure Oct 20, 2009
yacy.yellow performance setting for remote indexing configuration and latest chan… Jul 22, 2005

README.md

YaCy

Build Status

Deploy

What is this?

YaCy is a search engine software. It takes a new approach to search because it does not use a central server. Instead, its search results come from a network of independent peers. In such a distributed network, no single entity decides what gets listed, or in which order results appear.

The YaCy search engine runs on each user's own computer. Search terms are hashed before they leave the user's computer. Different from conventional search engines, YaCy is designed to protect the users' privacy. A user's computer can create with YaCy its individual search indexes and rankings, so that results better match what the user is looking for over time. YaCy also makes it easy to create a customized search portal with a few clicks.

Each YaCy user is either part of a large search network (YaCy contains a peer-to-peer network protocol to exchange search indexes with other YaCy search engine installations) or the user runs YaCy to produce a personal search portal that can be either public or private.

YaCy search portals can also be placed in intranet environment which makes YaCy a replacement for commercial enterprise search solutions. A network scanner makes it easy to discover all available http, ftp and smb servers.

To create a web index, YaCy has a web crawler for everybody, without censorship and central data retention:

  • search the web (automatically using all other YaCy peers)
  • co-operative crawling; support for other crawlers
  • intranet indexing and search
  • set up your own search portal
  • all users have equal rights
  • comprehensive concept to anonymise the users' index

To be able to perform a search using the YaCy network, every user has to set up their own node. More users are leading to higher index capacity and better distributed indexing performance.

License

YaCy is published under the GPL v2 The source code is inside the release package (see /source and /htroot).

Where is the documentation?

Documentation can be found at:

Every of these locations has a (YaCy) search functionality which combines all these locations into one search result.

Dependencies? What other software do I need?

You need Java 1.8 or later to run YaCy, nothing else (Java 1.7 can still be used to run the main 1.92/9000 release ) Please download it from https://www.java.com

YaCy also runs on Iced Tea 3. See https://icedtea.classpath.org

NO OTHER SOFTWARE IS REQUIRED! (you don't need apache, tomcat or mysql or whatever)

How do I start this software?

Startup and Shutdown of YaCy:

  • on GNU/Linux and OpenBSD: to start: execute ./startYACY.sh to stop : execute ./stopYACY.sh

  • on Windows: to start: double-click startYACY.bat to stop : double-click stopYACY.bat

  • on Mac OS X: please use the Mac Application and start or stop it like any other Mac Application (double-click to start)

How do I use this software, where is the administration interface?

YaCy is a build on a web server. After you started YaCy, start your browser and open

http://localhost:8090

There you can see your personal search and administration interface.

What if I install YaCy (headless) on a server?

You can do that but YaCy authorizes users automatically if they access the server from the localhost. After about 10 minutes a random password is generated and then it is not possible to log in from a remote location. If you install YaCy on a server that is not your workstation, then you must set an administration account immediately after the first start-up. Open:

http://:8090/ConfigAccounts_p.html

and set an administration account.

Can I run YaCy in a virtual machine or a container?

YaCy runs fine in virtual machines managed by software such as VirtualBox or VMware.

Container technology may be more flexible and lightweight and also works fine with YaCy.

These technologies can either be deployed locally, on remote machines you own, or in the 'cloud'. Decide what fits the most your privacy requirements.

Docker

Deploy easily YaCy on a Docker cloud provider of your choice (can be a machine you own) with the deploy button at the top of this page.

More details for YaCy with Docker in docker/Readme.md.

Heroku

Deploy easily on Heroku PaaS (Platform as a service) provider using the deploy button at the top.

More details for YaCy on Heroku in Heroku.md.

Port 8090 is bad, people are not allowed to access that port

You can forward port 80 to 8090 with iptables: iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8090 On some operation systems, you must first enable access to the ports you are using like: iptables -I INPUT -m tcp -p tcp --dport 8090 -j ACCEPT

How can I scale this; how much ram is needed; disk space?

YaCy can scale up to many millions of web pages in your own search index. The default assignment of RAM is 600MB which is assigned to the java process but not permanently used by it. The GC process will free the memory once in a while. If you have a small index (i.e. about 100000 pages) then you may assign less memory (i.e. 200MB) but if your index scales up to over 1 million web pages then you should start to increase the memory assignment. Open http://localhost:8090/Performance_p.html and set a higher/lower memory assignment. If you have millions of web pages in your search index then you might have gigabytes of disk space allocated. You can reduce the disk space i.e. setting the htcache space to a different size; to do that open http://localhost:8090/ConfigHTCache_p.html and set a new size.

Join the development!

YaCy was created with the help of many. About 30 programmers have helped, a list of some of them can be seen here: https://yacy.net/en/Join.html Please join us!

How to get the source code and how to compile YaCy yourself?

The source code is inside every YaCy release. You can also get YaCy from https://github.com/yacy/yacy_search_server by cloning the repository

git clone https://github.com/yacy/yacy_search_server

Please clone our code and help with development! The code is licensed under the GPL v2.

Compiling YaCy:

  • you need Java 1.8 or later and Apache Ant
  • just compile: "ant clean all" - then you can "./startYACY.sh" or "./startYACY.bat"
  • create a release tarball: "ant dist"
  • create a Mac OS release: "ant distMacApp" (works only on a Mac)
  • create a debian release: "ant deb"
  • work with eclipse: within eclipse you also need to start the ant build process because the servlet pages are not compiled by the eclipse build process after the dist procedure, the release can be found in the RELEASE subdirectory

Build with Maven:

  • for the first time goto subdirectory libbuild (which contains the maven parent pom)
  • compile with "mvn clean install -DskipTests", this will create all needed modules
  • after above you can use just the pom in the main directory to build YaCy with maven

Are there any APIs or how can I attach software at YaCy?

There are many interfaces build-in in YaCy and they are all based on http/xml and http/json. You can discover these interfaces if you notice the orange "API" icon in the upper right of some web pages in the YaCy web interface. Just click on it and you will see the xml/json version of the information you just have seen at the web page. A different approach is the usage of the shell script provided in the /bin subdirectory. The just call also the web interface pages. By cloning some of those scripts you can create more shell api access methods yourself easily.

Contact

Our primary point of contact is the German forum at http://forum.yacy.net We encourage you to start a YaCy forum in your own language.

If you have any questions, please do not hesitate to contact the maintainer: Send an email to Michael Christen (mc@yacy.net) with a meaningful subject including the word 'yacy' to prevent that your email gets stuck in my anti-spam filter.

If you like to have a customized version for special needs, feel free to ask the author for a business proposal to customize YaCy according to your needs. We also provide integration solutions if the software is about to be integrated into your enterprise application.

Germany, Frankfurt a.M., 26.11.2011 Michael Peter Christen