german presage

[matzgewinn]

I like to have a german presage. In openrepos I've read, tit needs a text corpus for that. I will try to get one. Do I have to take care for something special?
best regards

[rinigus]

very good, thanks for looking into it! Corpus has to be a plain text file that will be processed by the software. For reference, used text files were from 200MB - 2GB so far.

[matzgewinn]

I'm a bit confused of all the xml and so on formats of the german corpora.
So far I have two wordlists, they have a much smaller size than you wrote and are from here:
https://github.com/solariz/german_stopwords
and here:
https://sourceforge.net/projects/germandict/
They are plain text files - is this enough, and if yes how to proceed?

[rinigus]

No, we are looking for a text with words in sentences. Such as a collection of articles, texts of speeches, logs of chatrooms, movie subtitles. Something where you could calculate what's a chance to see some word after combination of other words. Just dictionary will not help in this case

…

[matzgewinn]

ok, understand. Now I have found four files here:
https://wortschatz.uni-leipzig.de/de/download
They are from articles, the web, wikipedia. Fils are 55mb, 118mb, 114mb and 131mb. Unfortunately the lines isn the files are counted, so there is a number before each sentence.

[rinigus]

should be simple to write a script removing the count lines and then process it by presage. So, that's already good

[matzgewinn]

Maybe I could simply delete all number-characters in the files - but that would probably corrupt some of the sentences... -? I don't think I have the skills to write a script :-/
Anyway - should I upload the files here or what is the further procedure?
Best regards!

[rinigus]

Thanks for finding the corpus files!

No, you cannot upload files over here. The best you could do is to make a list with specific links for the files we are expected to import.

As you don't know how to write a script, we'll have to wait till either me or someone else has time to look into it. Can't promise any specific time on my side, but if nobody will volunteer, I'll look into it.

Let's just use this issue to coordinate the effort and warn others if someone starts working on it.

So, please make a list of direct links to the files we need to download to get texts. If its using that format with one number in front of the sentence, so be it. That we will take into account.

[matzgewinn]

Thanks for your patience!
I at least could delete the numbers (by using the "sed" command), I hope the files are usable for you.
Here is the link, it is valid for seven days from now:
https://send.firefox.com/download/f9cfb13f4adb6d6a/#X7qFjSYId7PpfVkRcnmWGw

Let me know if there is something else I can do. I know its a little hard for someone experienced like You to handle "unprofessionals" like me, but I really like to learn and contribute - so sorry for any inconvenience.

[rinigus]

I am sorry, I completely forgot about it. Would you mind to resend it?

[matzgewinn]

Made one file out if it and deleted all numbers. hope you can use it. best regards!
https://send.firefox.com/download/1ef75eda68a7e53d/#oZpe7VFklKPXPdtJs52y1w

[rinigus]

Hmm, strange - link has expired again.

[matzgewinn]

ok, next try, if this fails too, I will use a differnet service :-)
https://send.firefox.com/download/4bf05f4b2c3e44e8/#NKsjqPx2kLK0R51xFo-S5w

[rinigus]

I have just pushed German packages to OpenRepos. Please test and report back.

[matzgewinn]

Thank You so much. Tested it for "normal" purpose like mails and sms, notes an so on, and its just working fine I think. A friend of mine who is used to androids text prediction was totally ok with it, too!
so lets hear what other people say...

[rinigus]

Excellent! I am closing it here. Please feel free to comment later in closed issue or open a new one if needed.

[matzgewinn]

Thank you again for your excellent work - if I can pay you a cup of coffee, let me know.

[rinigus]

No worries, main thing was to refresh the memory on how it was done. If you wish to donate, sure there is a link out there at github and easy to find by "donate rinigus", as far as I can see. But please feel free to skip it ...

Glad it worked out rather easily.