N-Gram

An N-gram is a sequence of N words: a 2-gram (or bigram) is a two-word sequence of words like “lütfen ödevinizi”, “ödevinizi çabuk”, or ”çabuk veriniz”, and a 3-gram (or trigram) is a three-word sequence of words like “lütfen ödevinizi çabuk”, or “ödevinizi çabuk veriniz”.

Smoothing

To keep a language model from assigning zero probability to unseen events, we’ll have to shave off a bit of probability mass from some more frequent events and give it to the events we’ve never seen. This modification is called smoothing or discounting.

Laplace Smoothing

The simplest way to do smoothing is to add one to all the bigram counts, before we normalize them into probabilities. All the counts that used to be zero will now have a count of 1, the counts of 1 will be 2, and so on. This algorithm is called Laplace smoothing.

Add-k Smoothing

One alternative to add-one smoothing is to move a bit less of the probability mass from the seen to the unseen events. Instead of adding 1 to each count, we add a fractional count k. This algorithm is therefore called add-k smoothing.

Video Lectures

For Developers

You can also see Python, Cython, Java, C, C++, Swift, Js, or C# repository.

Requirements

• Php 8.4 or higher
• Git

Php

To check if you have a compatible version of Php installed, use the following command:

php -V

You can find the latest version of Php here.

Git

Install the latest version of Git.

Download Code

In order to work on code, create a fork from GitHub page. Use Git for cloning the code to your local or below line for Ubuntu:

git clone <your-fork-git-link>

A directory called NGram will be created. Or you can use below link for exploring the code:

git clone https://github.com/starlangsoftware/NGram-Php.git

Open project with PhpStorm IDE

Steps for opening the cloned project:

• Start IDE
• Select File | Open from main menu
• Choose NGram-Php file
• Select open as project option
• Couple of seconds, dependencies will be downloaded.

For Contibutors

composer.json file

1. autoload is important when this package will be imported.

  "autoload": {
    "psr-4": {
      "olcaytaner\\WordNet\\": "src/"
    }
  },

2. Dependencies should be maximum (not only direct but also indirect references should also be given), everything directly in the code should be given here.

  "require-dev": {
    "phpunit/phpunit": "11.4.0",
    "olcaytaner/dictionary": "1.0.0",
    "olcaytaner/xmlparser": "1.0.1",
    "olcaytaner/morphologicalanalysis": "1.0.0"
  }

Data files

1. Add data files to the project folder. Subprojects should include all data files of the parent projects.

Php files

1. Do not forget to comment each function.

    /**
     * Returns true if specified semantic relation type presents in the relations list.
     *
     * @param SemanticRelationType $relationType element whose presence in the list is to be tested
     * @return bool true if specified semantic relation type presents in the relations list
     */
    public function containsRelationType(SemanticRelationType $relationType): bool{
        foreach ($this->relations as $relation){
            if ($relation instanceof SematicRelation && $relation->getRelationType() == $relationType){
                return true;
            }
        }
        return false;
    }

2. Function names should follow caml case.

    public function getRelation(int $index): Relation{

3. Write getter and setter methods.

    public function getOrigin(): ?string
    public function setName(string $name): void

4. Use standard javascript test style by extending the TestCase class. Use setup when necessary.

class WordNetTest extends TestCase
{
    private WordNet $turkish;

    protected function setUp(): void
    {
        ini_set('memory_limit', '450M');
        $this->turkish = new WordNet();
    }

    public function testSize()
    {
        $this->assertEquals(78327, $this->turkish->size());
    }

5. Enumerated types should be declared with enum.

enum CategoryType
{
    case MATHEMATICS;
    case SPORT;
    case MUSIC;
    case SLANG;
    case BOTANIC;

6. If there are multiple constructors for a class, define them as constructor1, constructor2, ..., then from the original constructor call these methods.

    public function constructor1(string $path, string $fileName): void
    public function constructor2(string $path, string $extension, int $index): void
    public function __construct(string $path, string $extension, ?int $index = null)

7. Use __toString method if necessary to create strings from objects.

    public function __toString(): string

8. Use xmlparser package for parsing xml files.

  $doc = new XmlDocument("../test.xml");
  $doc->parse();
  $root = $doc->getFirstChild();
  $firstChild = $root->getFirstChild();