Skip to content

Review cards by reading text and some other small changes #9

Closed
wants to merge 133 commits into from

2 participants

@dayjaby
dayjaby commented Jun 9, 2015

I made quite a lot of changes, without separating them well in different commits. I hope it doesn't become too troublesome.

First of all, I reverted some code of the last pull request. Instead of having a {translation} tag, we have now {line}, {t0},{t1},{t2} and so on. {t0} is the first tab-separated field in the line, {t1} the second one and so on. Apart from that I added a tag {filename}, which contains the path of the opened file.

Instead of preventing users to not add a note for a vocabulary that exists already, he has the option to overwrite it now. Overwritting is done by clicking on the blue '+'-button. In this case, all fields in the existing notes are overwritten. Except for the case that the name of a model's field ends with '+'. In this case field's new value is added to the old value with '
' in-between.

Usage example: My model has a field 'Examples+' with the value '{line} (<a href="file:///{filename}">File</a>)'. The Examples+ for the note 幸い (in Matthew 5) would look like this:
3 「心の貧しい人たちは、幸いである、天国は彼らのものである。 (File)
4 悲しんでいる人たちは、幸いである、彼らは慰められるであろう。 (File)
5 柔和な人たちは、幸いである、彼らは地を受けつぐであろう。 (File)

Added a support for saving files.

So far Yomichan handles 々 occurences not at all. Examples: 人々,様々. This is now handled during the sanitizing. It merely copies the previous kanji.

Added a preference option to immediately 'unlock' vocabulary. The meaning is that when I add a new vocabulary to my deck, it shall be put immediately out of the new queue into the learning queue.

These were the minor changes. The major change is that it's now possible to review Anki cards merely by reading the text with Yomichan. While reading, just lookup the definitions of words as you are used to do. This way, the vocabulary is added to the 'wrong' list. After completing a text, you click on the button 'Learn vocabulary'. It gives you a notification like "189 correct and 17 wrong". Every text will have a vocabulary list. Whenever you add or overwrite a note, it gets added to the vocabulary list. On saving a text, the vocabulary list and the text are separated by a line "### VOCABULARY IN THIS TEXT ###". I uploaded an example here.

Because we read a lot of undue cards, whereas Anki doesn't allow early reviews of them (or rather: Anki would calculate the new times as if the card is due), I made a custom scheduler, which handles these cases properly. For example if I read today one text with the word 私 and another text with 私 the following happens: After reviewing text 1 私 is rescheduled. After text 2 私's times aren't changed, because interval time = (due date - today). If (due date - today) is half of the interval time (the vocabulary is half-due), then the change is 50% of what it used to be in Anki.

Ok, that's all!

I consider this to be quite a revoluationary approach to vocabulary learning. A further enhancement would be to have a list of text files with the amount of due cards in each (where half-due cards are counted as 0.5). That way a student can focus on re-reading texts with a lot of due cards.

and others added some commits Nov 8, 2013
@FooSoft Adding stub for new deinflector and translator 9fa5a50
@FooSoft Adding header to compiler script ef234fb
@FooSoft Updating build scripts f8850bb
@FooSoft Dictionary can now search for terms and characters 5ec68e0
@FooSoft Updating the way tags are stored in dictionary 1a8ecef
@FooSoft Allow wildcard searches for terms 36a1031
@FooSoft Changing the way radicals are stored in the database 11daa98
@FooSoft Adding visual character search to dictionary 6d98f5b
@FooSoft Improved index generation for dictionary ad93c9a
@FooSoft Splitting tags before returning them from dictionary fb659e1
@FooSoft Adding parent deinflection type f737c0b
@FooSoft Switching deinflect.json to regex format 113cb48
@FooSoft Deinflector now properly working c77faa9
@FooSoft Fixes to deinflector 6b7e2b5
@FooSoft Renaming deinflection parameter term to source 617fa8a
@FooSoft Fixing deinflection bugs 6b5776a
@FooSoft More deinflector cleanup 19c5459
@FooSoft Adding translation engine 712e858
@FooSoft Adding code to make Yomichan work with the new dictionary and deinfle…
…ctor
dc63a28
@FooSoft Deleting old Japanese module 2bfac24
@FooSoft Moving new Japanese module in place of the old one 14e7993
@FooSoft Minor cleanup 362db52
@FooSoft Switching dictionary to return values in python dictionaries instead …
…of tuples
3f0682e
@FooSoft Renaming columns in database, making sure P tag gets parsed 0794e83
@FooSoft Renaming definitions to glossary fc5361e
@FooSoft Translator sorts popular terms before others e278bfa
@FooSoft Cleaning up reader_util, making sorting order better for translator 5597695
@FooSoft Switching way markup fields are formatted for cards e97bb12
@FooSoft Small fixups and tweaks 8de377f
@FooSoft Adding additional encodings for autodetect, fixing line wrap bug c8c98e4
@FooSoft Fixing up headers 98e4903
@FooSoft Moving location of deinflect and dictionary data 148b64a
@FooSoft Revert "Moving location of deinflect and dictionary data"
This reverts commit 148b64a.
7e8125b
@FooSoft Changling location of deinflect and dictionary data files 9e5ec9f
@FooSoft Updating UI for new features 919f207
@FooSoft Updating main window to function with new UI 1f8103a
@FooSoft Moving edict, kanjidic, kradfile data b600bb1
@FooSoft Adding default configuration, updating labels in preference dialog 37871a4
@FooSoft Updating preference system a78921b
@FooSoft More updates to preferences, getting closer to making it work edf2bba
@FooSoft Preferences partially working now 1d8395b
@FooSoft Preferences are now correctly updated 9b7f2bb
@FooSoft Reader now works with multiple profiles 8b63fc8
@FooSoft Cleanup 66ee265
@FooSoft Updating tags display on preferences ac5a427
@FooSoft Cleanup of helper functions for card generation 61748bf
@FooSoft Cleanup 2837f5b
@FooSoft Removing some constants 56c6577
@FooSoft Some basic support for displaying information about characters 895554d
@FooSoft Updating UI for preference dialog d5e16b0
@FooSoft Generate summary differently depending on profile 69909c0
@FooSoft Fixing word wrap not being applied correctly b059b2f
@FooSoft Cleanup 9810f65
@FooSoft Temporarily removing code for opening tar files. I'm going to add thi…
…s later after I figure out a better way to handle this logic.
b91c824
@FooSoft Cleaning up some stuff in dictionary and translator ca940dc
@FooSoft Correctly copy Kanji defintions to clipboard 505fff8
@FooSoft Updating Kanji lookup, temporarily disabling fancy search c6e298b
@FooSoft Look up Kanji at the same time as vocab when scanning text ec14972
@FooSoft Display Kanji results in order that they are encountered 5324fcf
@FooSoft Adding imporved support for searching using wildcards 95d3920
@FooSoft Updating UI labels 06c061d
@FooSoft Better handling of spaces and line breaks in files a3e47eb
@FooSoft Updating formatting for HTML output 9f3bb3a
@FooSoft Creating distribution script 8d87e0d
@FooSoft Adding handling for incorrect field markup 1f49616
@FooSoft Conditionally deleting old files in build scripts 49e2557
@FooSoft Improvements to text scanning and deinflection display 94bfd5c
@FooSoft Making search work without QString 6be55de
@FooSoft Fixing Yomichan loading the last file at the wrong offset 38cefbd
@FooSoft Incrementing version number d884fcb
@FooSoft Fixing bugs related to adding kana-only cards, bumping version 298b527
@FooSoft Don't open recent files on startup that don't exist anymore 89b6735
@FooSoft Entering text into vocab search field also displays results for Kanji 956e7df
@FooSoft Fixing relative paths in designer files ff9600a
@FooSoft Adding hotkeys for creating vocabulary and Kanji cards 6bdaba3
@FooSoft Making bracket keys move cursor around 1809d22
@FooSoft Bumping version number aa9eeab
@FooSoft Updating URLs a1e2966
@FooSoft Re-adding dictionary 5f1f013
@FooSoft Fixing unicode font names breaking Yomichan preferences 7763e7d
@FooSoft Removing unused include f623271
@FooSoft Fixing shell script 99eec38
@FooSoft Fixing shell script 6a90382
@FooSoft Adding UI for adding word list from Kindle deck d144d62
@FooSoft Stub for importing word list ee45d2c
@FooSoft Merge branch 'dev' of github.com:FooSoft/yomichan into dev e999db5
@FooSoft Can now import Kindle vocab decks 79282c1
@FooSoft Bumping version number f9a9d0b
@FooSoft Do not limit result listing when doing import 5570d52
@FooSoft Attempts to make the definition window not jump all over the place wh…
…en updating
c6aa7ec
@FooSoft Fixes c119b19
@FooSoft Updating UI e07cfab
@FooSoft Allow text word lists to be imported ac96ec7
@FooSoft Updating dictionary 67ecf48
@FooSoft Adding in contents of enamdict 2738889
@FooSoft Cleanup of HTML generation code b7080d6
@FooSoft Adding readme fe054ed
@FooSoft Updating feedback url 10ae7e2
@FooSoft Add checkbox to allow editing the main text area 67061d8
@FooSoft Add some initial text to the text area, text area is now editable by …
…default
9141971
@FooSoft Add nag to make people take the time to configure Yomichan d93128d
@FooSoft Version bump 95d27df
felixvd Context sentence properly cut off at newline
The newline character is taken into account 20 lines earlier when
looking for the start of the segment, but not when looking for its end.
When lines don’t end in punctuation (e. g. notes taken in lectures),
this could result in excessively long {sentence} fields.
8e25d2d
felixvd Basics for persistent text box functionality
Text is saved in the JSON upon closing and loaded on startup if the
flag “rememberTextBoxContent” is true (default). Text seems to be
escaped automatically for export into the JSON. Tested with quote
characters only.

Missing the UI for controlling this feature, as discussed.
77c4638
felixvd Reverting changes to getSentece() behavior
New line is now again ignored when looking for the end of the sentence.
a47014c
@FooSoft Minor fixups 7e6057d
@FooSoft Adding checkbox for rememberTextContent setting de85f3c
@FooSoft Improved update checking 66d9959
@FooSoft Version bump 52b4e20
@FooSoft Fixing bug where fields can get out of sync with Anki acea662
@FooSoft Fixing layout 3e010eb
@FooSoft Updating URLs 3d83385
@FooSoft Updating readme b04650a
@FooSoft Updating readme b9797df
@dayjaby dayjaby adding support for tab-separated translations a51c49d
@dayjaby dayjaby use of python std functions b914354
@FooSoft Merge pull request #8 from dayjaby/master
adding support for tab-separated translations
af98d57
@FooSoft OCD 2a0af04
@FooSoft Updating URL for updates.json aae37fd
@dayjaby dayjaby text files contain vocabulary lists, which can be learnt by one click f54ac43
@dayjaby dayjaby merged original repository 44eb202
@dayjaby dayjaby updated README 81620aa
@dayjaby dayjaby added filename to the fields, instead of overwriting fields they can …
…accumulate the content
be88d2f
@FooSoft
Owner
FooSoft commented Jun 10, 2015

Looks pretty sweet on a first glance. I will be looking over this over next couple of days to see if I can find any issues or anything to knitpick about : )

@FooSoft
Owner
FooSoft commented Jun 10, 2015

One small thing: I have a updates file that is used to automatically notify users about changes to Yomichan when they run the plugin: http://foosoft.net/projects/yomichan/dl/updates.json

If you can provide a concise feature list, that would be swell.

@dayjaby
dayjaby commented Jun 10, 2015

The biggest problem with this update is that a lot of things happen under the surface. A new user might accidently click on "Learn vocabulary" and non-expected things occur. Right now I use the "Anki" panel to display the vocabs of a text. But it's not clear that all these are actually learned after clicking on "Learn vocabulary". Furthermore there is no extra list of "bad" vocabulary. And the user should have the option to remove vocabulary out of the list.

Or on saving text files he might be surprised that there is some kind of vocabulary list attached. So maybe we could have a "Save..." menu item and a "Save file and vocab..." menu item to make it clear.

And overwriting hiragana-only vocabulary is not implemented yet (the one where you click on the small "+"). Edit: DONE

And if you lookup a vocabulary, not all vocabulary should be added to the bad list, but maybe only the longest match. So if you lookup 踏み抜く, then 踏む shouldn't be added to the bad list. And there should probably be a status message at the bottom like "踏み抜く has been put into the wrongly answered vocabulary set" Edit: DONE

Another critical point: The statistics of Anki are distorted heavily by the usage of this. At the moment each card is answered in 0 seconds. Something like "Average answer time: 0.4s (153.1 cards/minute)" contains no information at all. If we find a good way to time the vocabulary, then it would be possible to compare this method to the usual one. Like "Oh, now I only need 2 seconds per card instead of 10. That's awesome" ;)

Ok, so far my self-criticism.

@dayjaby
dayjaby commented Jun 14, 2015

With the newest commit, it's possible to store the text files in collection.media\Yomichan and these text files get automatically added to the deck list in Anki. That way it's possible to quickly decide which text contains a lot of due vocabulary. A benefit of using collection.media is that the text files can be synchronized as well!
Clicking on such a text deck causes Yomichan to open the corresponding file.

image

Some implementation notes:

  • Adding decks to the tree can be done with a custom Scheduler. But each text deck still needs an unique existing deck id. Thus new decks are created, but they remain empty.
  • Whenever the deck browser opens, all text files are parsed and the dueness is calculated. As before mentioned, cards don't have to be 100% due. A 50% card is considered as 0.5 due cards.
@FooSoft
Owner
FooSoft commented Jun 14, 2015

Coolness, I'm going over the changes locally and making a couple of small tweaks here and there.

@dayjaby
dayjaby commented Jun 14, 2015

Good, but make sure to use the latest commits. I tried to put everything in its proper place. For example the custom scheduler in anki_bridge.py instead of reader.py.

@FooSoft
Owner
FooSoft commented Jun 14, 2015

Yeah, I'm pulling from your master branch. A problem I found is that there is a bunch of Anki stuff inside of yomichan.py. One goal of the way Yomichan is set up is to keep all of the Anki related things abstracted inside of anki_bridge.py -- this way if and when version updates break Yomichan, it's possible to isolate the changes to just one module. Furthermore, everything non-Anki related should work fine just by running yomichan.py (basically running outside of Anki environment). Right now when I try this I get

alex@flatline /m/s/p/yomichan> ./yomichan.py 
Traceback (most recent call last):
  File "./yomichan.py", line 26, in <module>
    from aqt import mw
ImportError: No module named aqt
@dayjaby
dayjaby commented Jun 14, 2015

Fixed.

@FooSoft
Owner
FooSoft commented Jun 14, 2015

A couple of other things for consistency with how other things work in Yomichan:

  • Things on the Other tab should really be put onto Anki tab since they are related to Anki-specific functionality; Anki tab automatically gets disabled when running in standalone mode.
  • The info icons don't do anything when not running as plugins and should therefore not show up when doing lookups.
@FooSoft
Owner
FooSoft commented Jun 14, 2015

The checkbox label also appears to be cut off:
cutoff

@dayjaby
dayjaby commented Jun 15, 2015

Preferences window should be fine now. Overwrite icons are removed if running the standalone.

@dayjaby
dayjaby commented Jun 25, 2015

So what's the current status?

Atm I am writing some code that allows to "export vocabulary". Reading such a text results in having the necessary cards being automatically generated. That's practical if you want to share text with others.

Apart from that I started to work on Korean support as well. That's going to be in another pull request though ^^

@FooSoft
Owner
FooSoft commented Jun 25, 2015

It's a bunch changes, and I want to take the time to make sure I understand exactly how everything works before I merge. It's just taking me a little while cuz I have some stuff going on at school at the moment.

It will probably be another couple of weeks before I have a chance to really dig in ^^

Korean support sounds good, although it will be interesting trying to generalize the Japanese specific stuff scattered out throughout the program.

@dayjaby
dayjaby commented Jun 25, 2015

Yes, I think by adding another language, the Japanese stuff will get generalized out automatically.

@dayjaby
dayjaby commented Jun 26, 2015

As mentioned before, vocabulary is "exported" now. That means the entire definition of a vocabulary is placed into the text file now too.

Apart from that, upon opening a text file, any music/video/... file gets opened too, that has the same prefix. E.g. if I open AcchiKocchi1.txt, then AcchiKocchi1.mkv will be opened too. I did that, because most of my study materials that I use in Yomichan I got from some audio or video source. Having these available to accompany the reading is quite useful.

@dayjaby
dayjaby commented May 16, 2016

@FooSoft I've seen in your github activities that you started working on Yomichan again. I wondered if there is some common ground for collaboration, even though my previous commits deviate from Yomichan's original usage. Nonetheless, things like adding new languages, tiding up and modularizing previous code might be beneficial for both of us.

And I'd like to hear about your future plans :) If you'd like to discuss things in private, feel free to email me on dayjaby@gmail.com

@dayjaby dayjaby referenced this pull request May 26, 2016
Closed

Kanji iteration mark #14

@dayjaby dayjaby closed this Jun 9, 2016
@dayjaby
dayjaby commented Jun 9, 2016

Closed as it's improbable this is going to be merged in the near future.

@dayjaby dayjaby referenced this pull request Jun 9, 2016
Closed

Korean support #10

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.