Skip to content
This repository has been archived by the owner on Oct 15, 2022. It is now read-only.

Latest commit

 

History

History
629 lines (421 loc) · 29.1 KB

README.md

File metadata and controls

629 lines (421 loc) · 29.1 KB
Raw

DuckDuckHack Developer Overview

What is this?

DuckDuckGo is a general purpose search engine. We've created a platform called DuckDuckHack that enables developers to write open source plugins on top of the search engine (like add-ons for Firefox). DuckDuckGo plugins react to search queries and provide useful instant answers above traditional links.

DuckDuckHack is very much a work in progress. Some plugin types have better interfaces than others. We will be improving the platform based on your feedback.

Why should I make plugins?

We hope you will consider making DuckDuckGo plugins to:

##Overview

There are currently four types of DuckDuckGo plugins:

Goodies — calculations and cheat sheets.

Spice — external API calls.

Fathead — keyword databases.

Longtail — full-text data.

  • Examples: wikipedia, lyrics, stack overflow, etc.
  • Status: alpha
  • Languages: Perl, Node, Ruby, Python (maybe others)
  • Involves: formatting data sets to answer general queries.

How to follow the Documentation

This repo contains all plugin-agnostic information that you'll need. In the Getting Started section below, you'll find overviews and documentation trails for Spice and Goodie plugins. These two plugin types have more well-defined processes and you just have to follow the list of links in order to learn all you need to know. For Fathead and Longtail plugins, you should see their individual repositories for how-to instructions (linked in Getting Started). The zeroclickinfo-goodies and zeroclickinfo-spice repositories each contain more detailed information on their respective plugin types and their readme's are linked to from the tracks below. At the end of a section in one of these repos, there will always be a link that will take you back to this page and the documentation trail that you have been following.

Getting Started

Step 1.  Decide what you want to work on. If you don't have any ideas, start here.

Step 2.  Figure out your plugin type (see Plugin Types above). If the right type is not obvious, please ask us. Sometimes multiple plugin types could work, and we can help you figure out which one would work best. Consider the following when choosing what type of plugin to make.

Guidelines

  • DuckDuckGo plugins appear at the top of search results pages, which is a sacred space! Please follow these guidelines to ensure the quickest path to going live.

  • Use the right plugin type. If your plugin uses external APIs in real time, it should be Spice.

  • Better than links. Since instant answers are above the traditional links, they should be unambiguously better than them.

  • No false positives. A false positive is an irrelevant instant answer. Only return an instant answer when you know it is good, and otherwise return nothing.

  • Minimize vertical space. Only include the most important information and then offer the user to click through for more if needed.

  • Readable answers. If textual, create sentences or short statements that users can actually read.

  • Consistent design. When in doubt, copy what already exists or ask us!

Step 3.  Fork the right repository (GitHub instructions):

Step 4.  Now it's choose-your-own-adventure time!

  • For Goodies, check out the Goodies Overview below. This will give you a list of links to other pages that will guide you through the goodie process.

  • For Spice, proceed to the Spice Overview below. This section will walk you through everything you need to build a basic spice plugin.

  • For Fathead, check out the Readme in the fathead repository.

  • For Longtail, check out the Readme in the longtail repository.

Goodies Overview

Follow this list to go through the goodie progression.

  1. Basic Tutorial -- this will show you the fundamentals of making a plugin. It's a simple walkthrough-by-example and gives a good introduction to the system.
  2. Testing Triggers -- this will lead you through how to use duckpan, our command-line utility, to test the plugins that you've written and make sure your triggers are working properly.
  3. Submitting Plugins -- this section guides you through the plugin submission process, and is the last section that you need to gain a basic understanding of the entire process.
  4. Once you're familiar with the above three sections, it's time to move on the Goodies repository, which contains information about advanced goodie creation, and the Advanced section of the plugin-agnostic docs (this page).

Spice Overview

Follow this list to go through the spice progression.

  1. Basic Tutorial -- this will show you the fundamentals of making a plugin. It's a simple walkthrough-by-example and gives a good introduction to the system.
  2. Spice Handle Functions -- this section provides an overview of the different variables that a spice plugin can process.
  3. Testing Triggers -- this will lead you through how to use duckpan, our command-line utility, to test the plugins that you've written and make sure your triggers are working properly.
  4. Spice Callback Functions -- this section explains how JavaScript callback functions are generated by the plugin system.
  5. Testing Spice -- this section introduces you to the spice testing process.
  6. Submitting Plugins -- this section guides you through the plugin submission process, and is the last section that you need to gain a basic understanding of the entire process.
  7. Once you're familiar with the above sections, it's time to move on the Spice repository, which contains an advanced section with information about more involved spice creation, and the Advanced section of the plugin-agnostic docs (this page).

Basic Tutorial

In this tutorial, we'll be making a Goodie plugin that checks the number of characters in a given search query. The end result will look like this and works like this. The same framework is used to trigger Spice plugins.

Let's begin. Open a text editor like gedit, notepad or emacs and type the following:

package DDG::Goodie::Chars;
# ABSTRACT: Give the number of characters (length) of the query.

Each plugin is a Perl package, so we start by declaring the package namespace. In a new plugin, you would change Chars to the name of the new plugin (written in CamelCase format).

The second line is a special comment line that gets parsed automatically to make nice documentation (by Dist::Zilla).

Next, type the following use statement to import the magic behind our plugin system.

use DDG::Goodie;

A Note on Modules

Right after the above line, you should include any Perl modules that you'll be leveraging to help generate the answer. Make sure you add those modules to the dist.ini file in this repository. If you're not using any additional modules, carry on!

Now here's where it gets interesting. Type:

triggers start => 'chars';

triggers are keywords that tell us when to make the plugin run. They are trigger words. When a particular trigger word is part of a search query, it tells DuckDuckGo to trigger the appropriate plugins.

In this case there is one trigger word: chars. Let's say someone searched "chars this is a test." chars is the first word so it would trigger our Goodie. The start keyword says, "Make sure the trigger word is at the start of the query." The => symbol is there to separate the trigger words from the keywords (for readability).

Now type in this line:

handle remainder => sub {

Once triggers are specified, we define how to handle the query. handle is another keyword, similar to triggers.

You can handle different aspects of the search query, but the most common is the remainder, which refers to the rest of the query (everything but the triggers). For example, if the query was "chars this is a test", the trigger would be chars and the remainder would be this is a test.

Now let's add a few more lines to complete the handle function.

handle remainder => sub {
    return 'Chars: ' . length $_ if $_;
    return;
};

This function (the part within the {} after sub) is the meat of the Goodie. It generates the instant answer that is displayed at the top of the search results page.

Whatever you are handling is passed to the function in the $_ variable ( $_ is a special default variable in Perl that is commonly used to store temporary values). For example, if you searched DuckDuckGo for "chars this is a test", the value of $_ will be "this is a test", i.e. the remainder.

Let's take a closer look at the first line of the function.

return 'Chars: ' . length $_ if $_;

The heart of the function is just this one line. The remainder is in the $_ variable as discussed. If it is not blank ( if $_ ), we return the number of chars using Perl's built-in length function.

Perl has a lot of built-in functions, as well as thousands and thousands of modules available via CPAN. You can leverage these modules when making Goodies, similar to how the Roman Goodie uses the Roman module.

If we are unable to provide a good instant answer, we simply return nothing. And that's exactly what the second line in the function does.

return;

This line is only run if $_ contained nothing, because otherwise the line before it would return something and end the function.

Now, below your function type the following line:

zci is_cached => 1;

This line is optional. Goodies technically return a ZeroClickInfo object (abbreviated as zci). This effect happens transparently by default, but you can override this default behavior via the zci keyword.

We set is_cached to true (0 is false, 1 is true) because this plugin will always return the same answer for the same query. This speeds up future answers by caching them (saving previous answers).

Finally, all Perl packages that load correctly should return a true value so add a 1 on the very last line.

1;

And that's it! At this point you have a working DuckDuckHack Goodie plugin. It should look like this:

package DDG::Goodie::Chars;
# ABSTRACT: Give the number of characters (length) of the query.

use DDG::Goodie;

triggers start => 'chars';

handle remainder => sub {
    return 'Chars: ' . length $_ if $_;
    return;
};

zci is_cached => 1;

1;

Review

The plugin system works like this at the highest level:

  • We break the query (search terms) into words. This process happens in the background.

  • We see if any of those words are triggers (trigger words). These are provided by each of the plugins. In the example, the trigger word is chars.

  • If a Goodie plugin is triggered, we run its handle function.

  • If the Goodie's handle function outputs an instant answer via a return statement, we pass it back to the user.

Where to go from here

Click to return to the Goodies Overview or the Spice Overview.

Plugin-agnostic Information

Testing Triggers

Before reading this section, make sure you've at least worked through the basic tutorial.

Step 1.  Install our DuckDuckHack utility called duckpan:

curl http://duckpan.org/install.pl | perl

This script will setup local::lib, which is a way to install Perl modules without changing your base Perl installation. (If you already use local::lib or perlbrew, don't worry, this script will intelligently use what you already have.)

If you didn't have a local::lib before running the install script, you will need to run the script twice. It should tell you when like this:

please now re-login to your user account and run it again!

If everything works, you should see this at the end:

EVERYTHING OK! You can now go hacking! :)

Note that with local::lib now installed, you can easily install Perl modules with cpanm.

cpanm App::DuckPAN
App::DuckPAN is up to date.

Step 2.  Go to your fork of the repository (a directory or folder on your computer).

cd zeroclickinfo-goodies/

Step 3.  Install the repository requirements using duckpan.

duckpan installdeps

This command will install all the Perl modules used by the DuckDuckGo plugins within your local repository. These requirements are defined in the /dist.ini file (at the root).

Step 4. Add your plugin.

Make a new file in the lib/DDG/Goodie/ directory for Goodies or the lib/DDG/Spice/ directory for Spice. The name of the file is the name of the plugin followed by the extension .pm because it is a Perl package. For example, if the name of your plugin was TestPlugin, the file would be TestPlugin.pm.

Step 5. Test your trigger(s) interactively.

Type this command at the command line.

duckpan query

First, this command will output all of the plugins available in your local plugin repository.

Using the following DDG::Goodie plugins:

 - DDG::Goodie::Xor (Words)
 - DDG::Goodie::SigFigs (Words)
 - DDG::Goodie::EmToPx (Words)
 - DDG::Goodie::Length (Words)
 - DDG::Goodie::ABC (Words)
 - DDG::Goodie::Chars (Words)
 ...

You should see your plugin in there as well. When the output is finished it gives you an interactive prompt.

(Empty query for ending test)
Query:

Now you can type in any query and see what the response will be.

Query: chars this is a test

DDG::ZeroClickInfo  {
    Parents       WWW::DuckDuckGo::ZeroClickInfo
    Linear @ISA   DDG::ZeroClickInfo, WWW::DuckDuckGo::ZeroClickInfo, Moo::Object
    public methods (3) : is_cached, new, ttl
    private methods (0)
    internals: {
        answer   14,
        answer_type   "chars",
        is_cached   1
    }
}

There is a lot of debugging output, but you will want to pay special attention to the internals section.

    internals: {
        answer   14,
        answer_type   "chars",
        is_cached   1
    }

Here you can see the answer returned, as well as any zci keywords (by default there will be a default answer_type and is_cached value).

Simply hit enter (a blank query) to exit interactive mode.

Query:

\_o< Thanks for testing!

Where to go now:

Click to return to the Goodies Overview or the Spice Overview.

Submitting Plugins

Step 1.  Commit your changes.

git commit -a -m "My first plugin that does X is ready to go!"

Step 2.  Get your commit history how you like it.

git rebase -i origin/master

Step 3.  Push your forked repository back to GitHub.

git push

Step 4.  Go into GitHub and submit a pull request! That will let us know about your plugin and start the conversation about integrating it into the live search engine.

Where to go now:

You're pretty much done! Now it's time to learn the advanced stuff! Click to return to the Goodies Overview or the Spice Overview. Each of these sections has links to follow so you can read up on the more advanced facets of plugin development.

===

Advanced

Advanced Triggers

In the Basic tutorial we walked through a one word trigger and in the Spice handle functions section we walked through a simple regexp trigger.

Here are some more advanced trigger techniques you may need to use:

Multiple trigger words.  Suppose you thought that in addition to chars, numchars should also trigger the Chars Goodie. You can simply add extra trigger words to the triggers definition.

triggers start => 'chars', 'numchars';

Trigger locations.  The keyword after triggers, start in the Chars example, specifies where the triggers need to appear. Here are the choices:

  • start - just at the start of the query
  • end - just at the end of the query
  • startend - at either end of the query
  • any - anywhere in the query

Combining locations.  You can use multiple locations like in the Drinks Spice.

triggers any => "drink", "make", "mix", "recipe", "ingredients";
triggers start => "mixing", "making";

Regular Expressions.  As we walked through in the Spice handle functions section you can also trigger on a regular expression.

triggers query_lc => qr/^@([^\s]+)$/;

We much prefer you use trigger words when possible because they are faster on the backend. However, in some cases regular expressions are necessary, e.g. when you need to trigger on sub-words.

Regexp types.  Like trigger words, regular expression triggers have several keywords as well. In the above example query_lc was used, which operates on the lower case version of the full query. Here are the choices:

  • query_raw - the actual (full) query
  • query - with extra whitespace removed
  • query_lc - lower case version of the query and extra whitespace removed
  • query_clean - lower case with non alphanumeric ASCII and extra whitespace removed
  • query_nowhitespace - with whitespace totally removed
  • query_nowhitespace_nodash - with whitespace and dashes totally removed

If you want to see some test cases where these types are enumerated check out our internal test file that tests they are generated properly.

Two-word+ triggers  Right now trigger words only operate on single words. If you want to operate on a two or more word trigger, you have a couple of options.

triggers query_lc => qr/cost of living/;
  • Use single word queries and then further qualify the query within the handle function as explained in the Advanced handle functions section.

Advanced Handle Functions

In the Basic tutorial we walked through a simple query transformation and in the Spice handle functions section we walked through a simple return of the query.

Here are some more advanced handle techniques you may need to use:

Further qualifying the query.  Trigger words are blunt instruments; they may send you queries you cannot handle. As such, you generally need to further qualify the query (and return nothing in cases where the query doesn't really qualify for your goodie).

There are number of techniques for doing so. For example, the first line of Base Goodie has a return statement paired with unless.

return unless  /^([0-9]+)\s*(?:(?:in|as)\s+)?(hex|hexadecimal|octal|oct|binary|base\s*([0-9]+))$/;

You could also do it the other way, like the GoldenRatio Goodie.

if ($input =~ /^(?:(?:(\?)\s*:\s*(\d+(?:\.\d+)?))|(?:(\d+(?:\.\d+)?)\s*:\s*(\?)))$/) {

Another technique is to use a hash to allow specific query strings, as the GUID Goodie does.

my %guid = (
    'guid' => 0,
    'uuid' => 1,
    'globally unique identifier' => 0,
    'universally unique identifier' => 1,
    'rfc 4122' => 0,
    );

return unless exists $guid{$_};

Handling the whole query.  In the Chars example, we handled the remainder. You can also handle:

  • query_raw - the actual (full) query
  • query - with extra whitespace removed
  • query_parts - like query but given as an array of words
  • query_nowhitespace - with whitespace totally removed
  • query_nowhitespace_nodash - with whitespace and dashes totally removed

For example, the Xor Goodie handles query_raw and the ABC Goodie handles query_parts.

Using files.  You can use simple text/html input files for display or processing.

my @words = share('words.txt')->slurp;

The Passphrase Goodie does this for processing purposes and the PrivateNetwork Goodie does it for display purposes.

The files themselves go in the /share/goodie/ directory.

Generating data files. You may also need to generate data files. If you do so, please also include the generation scripts. These do not have to be done in Perl, and you can also put them within the /share/goodie/ directory. For example, the CurrencyIn Goodie uses a Python script to generate the input data.

There are a couple more sections on advanced handle techniques depending on Plugin type:

Advanced Testing

The testing triggers section explained interactive testing. Before going live we also make programmatic tests for each plugin.

Step 1.  Add your plugin test file.

Make a new file in the test directory t/. The name of the file is the name of your plugin, but this time followed by the extension .t for test because it is a Perl testing file. For example, if the name of your plugin was TestPlugin, the file would be TestPlugin.t.

The top of the file reads like a normal Perl script with some use statements to include testing modules, including the DuckDuckGo testing module.

#!/usr/bin/env perl

use strict;
use warnings;
use Test::More;
use DDG::Test::Goodie;

Then you define any default zci values that you set in your plugin.

zci answer_type => 'chars';
zci is_cached => 1;

These should match exactly what you set in your .pm file.

Next comes the actual testing function.

ddg_goodie_test(
        [qw(
                DDG::Goodie::Chars
        )],
        'chars test' => test_zci('Chars: 4'),
        'chars this is a test' => test_zci('Chars: 14'),
);

For each test, you include a line like this:

        'chars test' => test_zci('Chars: 4'),

The first part, 'chars test' in this example, is the test query. The second part, test_zci('Chars: 4') calls the test function and checks if Chars: 4 is the answer.

Finally you end a testing file with this line.

done_testing;

The full file should look like this:

#!/usr/bin/env perl

use strict;
use warnings;
use Test::More;
use DDG::Test::Goodie;

zci answer_type => 'chars';
zci is_cached => 1;

ddg_goodie_test(
        [qw(
                DDG::Goodie::Chars
        )],
        'chars test' => test_zci('Chars: 4'),
        'chars this is a test' => test_zci('Chars: 14'),
);

done_testing;

Step 2.  Test your plugin programmatically.

Run your plugin test file like this:

perl -Ilib t/Chars.t

If successful, you should see a lot of ok lines.

ubuntu@yegg:~/zeroclickinfo-goodies$ perl -Ilib t/Chars.t
ok 1 - Testing query chars test
ok 2 - Testing query chars this is a test
1..2

If unsuccessful, you will see one or more not ok lines followed with some debugging output to help you chase down the error(s).

ok 1 - Testing query chars test
not ok 2 - Testing query chars this is a test
#   Failed test 'Testing query chars this is a test'
#   at /usr/local/ddg.cpan/perl5/lib/perl5/DDG/Test/Goodie.pm line 69.
#     Structures begin differing at:
#          $got->{answer} = '14'
#     $expected->{answer} = '15'
1..2
# Looks like you failed 1 test of 2.