InitialDescription

Initial Project Description

As explained in the class, we have a repository called jbs-text that contains many text "pieces" from the Jewish Bookshelf (JBS). The text is represented in Json format, where each Json object has at least an attribute called "uri" that holds the id of the text element, and an attribute called "jbo:text" that holds the textual content. You are invited to explore jbs-text a bit. It is recommended to install a Chrome plugin called ‪JSON Formatter for better presentation of Json files.

The basic task

This project deals with classification of text elements (Json objects) into one or more predefined topics. In the first Sprint (1-month iteration) of the project we will focus on the classification of "Psukim" (פסוקים) from the Tanach. This means that we will classify Json objects having a "uri" attribute of the form "jbr:tanach-x-y-z" (e.g. "jbr:tanach-1-1-1 that corresponds to the Pasuk (בראשית א א). For example, take a look at this file called tanach-1.json that holds all the psumik of sefer bereshit.

One possible topic is "עקדת יצחק". There are several psukim in sefer Bereshit that describe this topic (פרק כ"ב א-יט), and your task is to find them, automatically of course. More generally, you will be provided with a set of topics, and the task will be to "mark" psukim as belonging to one or more topics (of course, some of the psukim may not belong to any topic).

The set of topic is actually a set of Wikipedia pages (in Hebrew). For example, the topic "עקדת יצחק" is described by this Hebrew Wikipedia page. You should write code that analyzes the textual content of this page and hopefully returns the set of psukim that are related to the topic. Take a look at the bottom of the page in the "קטגוריות" section. Each Wikipedia's page belongs to one or more categories. Pressing on the "סיפורי ספר בראשית" category will reveal more potential topics.

What should be done in the exploration phase?

1. Go over (manually) many Tanach related "topics" and formulate (i.e., write down) a method for identifying the Psukim that are related to the topic. This method will be implemented in the first Sprint.
2. Download IntelliJ and create a Maven project for exploration purposes.
3. Your future system that you will develop in the Sprint will be provided with dozens of Wikipedia pages that represent topics from the Tanach, and in the first stage will extract the text from the Wikipedia articles for further processing. In the exploration phase you should choose the Java library to work with, for this task of text extraction. I know about a library called jsoup that allows to parse HTML pages. However I think that jsoup does not support XPATH queries, and perhaps a library that does support XPATH is better. You are expected to present a basic capability to extract text from a Wikipedia Hebrew page, preferably after reviewing more libraries. It is important to choose a library that is well known and supported by the community of developers.
4. The input to your future system and the output will be represented in Json format. Students in the previous semester were quite satisfied from working with the gson library. Add the gson library to the Maven and present basic capabilities to read a Json file and to produce an output Json file.

That's all for the exploration phase that should be completed until the 6/4.

Additional Notes:

• In Maven, there is a uniform way to add dependencies for the project. Use that way (it involves updating the pom.xml file with dependencies taken from Maven Central).
• All your code should be pushed to this GitHub repository. I recommend using Git for Windows which is a command line interface for git. Read about git basics here. You may find this visual tutorial helpful as well.
• Any questions related to the exploration phase should be asked in the special GitHub issue I opened for that. Don't forget to tag @omishali.