Unexpected char 65.533

[JhonnySalles]

I don't know if it's an error on my part or the dictionary files aren't working. In case it is my fault, I'm sorry, I didn't understand how dictionaries work, I think it was just load them to use the tokenizer.
I am trying to use in my program to generate phrases in anki for my studies.

When reading the "system_small.dic" or "system_core.dic" files, I get the following message in eclipse. I downloaded the files found on the home page, and performing a test by cmd I find the same error reported as shown in the image.

sudachi-dictionary-20200330-small.zip
sudachi-dictionary-20200330-core.zip

CMD:

Eclipse:

Exception in thread "JavaFX Application Thread" java.lang.IllegalArgumentException: javax.json.stream.JsonParsingException: Unexpected char 65.533 at (line no=1, column no=1, offset=0)
at sudachi@0.4.0/com.worksap.nlp.sudachi.Settings.parseSettings(Settings.java:115)
at sudachi@0.4.0/com.worksap.nlp.sudachi.JapaneseDictionary.buildSettings(JapaneseDictionary.java:97)
at sudachi@0.4.0/com.worksap.nlp.sudachi.JapaneseDictionary.(JapaneseDictionary.java:52)
at sudachi@0.4.0/com.worksap.nlp.sudachi.JapaneseDictionary.(JapaneseDictionary.java:48)
at sudachi@0.4.0/com.worksap.nlp.sudachi.DictionaryFactory.create(DictionaryFactory.java:47)
at TextosJapones/org.jisho.textosJapones.controller.TelaProcessarFrasesController.processaTexto(TelaProcessarFrasesController.java:217)
at TextosJapones/org.jisho.textosJapones.controller.TelaProcessarFrasesController.lambda$4(TelaProcessarFrasesController.java:454)
at javafx.base/com.sun.javafx.binding.ExpressionHelper$Generic.fireValueChangedEvent(ExpressionHelper.java:360)
at javafx.base/com.sun.javafx.binding.ExpressionHelper.fireValueChangedEvent(ExpressionHelper.java:80)
at javafx.base/javafx.beans.property.ReadOnlyBooleanPropertyBase.fireValueChangedEvent(ReadOnlyBooleanPropertyBase.java:72)
at javafx.graphics/javafx.scene.Node$FocusedProperty.notifyListeners(Node.java:8159)
at javafx.graphics/javafx.scene.Scene$12.invalidated(Scene.java:2158)
at javafx.base/javafx.beans.property.ObjectPropertyBase.markInvalid(ObjectPropertyBase.java:112)
at javafx.base/javafx.beans.property.ObjectPropertyBase.set(ObjectPropertyBase.java:147)
at javafx.graphics/javafx.scene.Scene$KeyHandler.setFocusOwner(Scene.java:4030)
at javafx.graphics/javafx.scene.Scene$KeyHandler.requestFocus(Scene.java:4077)
at javafx.graphics/javafx.scene.Scene.requestFocus(Scene.java:2125)
at javafx.graphics/javafx.scene.Node.requestFocus(Node.java:8320)
at javafx.controls/com.sun.javafx.scene.control.behavior.TextAreaBehavior.mousePressed(TextAreaBehavior.java:264)
at javafx.controls/javafx.scene.control.skin.TextAreaSkin$ContentView.lambda$new$0(TextAreaSkin.java:1201)
at javafx.base/com.sun.javafx.event.CompositeEventHandler$NormalEventHandlerRecord.handleBubblingEvent(CompositeEventHandler.java:218)
at javafx.base/com.sun.javafx.event.CompositeEventHandler.dispatchBubblingEvent(CompositeEventHandler.java:80)
at javafx.base/com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:238)
at javafx.base/com.sun.javafx.event.EventHandlerManager.dispatchBubblingEvent(EventHandlerManager.java:191)
at javafx.base/com.sun.javafx.event.CompositeEventDispatcher.dispatchBubblingEvent(CompositeEventDispatcher.java:59)
at javafx.base/com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:58)
at javafx.base/com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
at javafx.base/com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)
at javafx.base/com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
at javafx.base/com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)
at javafx.base/com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
at javafx.base/com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)
at javafx.base/com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
at javafx.base/com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)
at javafx.base/com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
at javafx.base/com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)
at javafx.base/com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
at javafx.base/com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)
at javafx.base/com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
at javafx.base/com.sun.javafx.event.BasicEventDispatcher.dispatchEvent(BasicEventDispatcher.java:56)
at javafx.base/com.sun.javafx.event.EventDispatchChainImpl.dispatchEvent(EventDispatchChainImpl.java:114)
at javafx.base/com.sun.javafx.event.EventUtil.fireEventImpl(EventUtil.java:74)
at javafx.base/com.sun.javafx.event.EventUtil.fireEvent(EventUtil.java:54)
at javafx.base/javafx.event.Event.fireEvent(Event.java:198)
at javafx.graphics/javafx.scene.Scene$MouseHandler.process(Scene.java:3862)
at javafx.graphics/javafx.scene.Scene.processMouseEvent(Scene.java:1849)
at javafx.graphics/javafx.scene.Scene$ScenePeerListener.mouseEvent(Scene.java:2590)
at javafx.graphics/com.sun.javafx.tk.quantum.GlassViewEventHandler$MouseEventNotification.run(GlassViewEventHandler.java:409)
at javafx.graphics/com.sun.javafx.tk.quantum.GlassViewEventHandler$MouseEventNotification.run(GlassViewEventHandler.java:299)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at javafx.graphics/com.sun.javafx.tk.quantum.GlassViewEventHandler.lambda$handleMouseEvent$2(GlassViewEventHandler.java:447)
at javafx.graphics/com.sun.javafx.tk.quantum.QuantumToolkit.runWithoutRenderLock(QuantumToolkit.java:411)
at javafx.graphics/com.sun.javafx.tk.quantum.GlassViewEventHandler.handleMouseEvent(GlassViewEventHandler.java:446)
at javafx.graphics/com.sun.glass.ui.View.handleMouseEvent(View.java:556)
at javafx.graphics/com.sun.glass.ui.View.notifyMouse(View.java:942)
at javafx.graphics/com.sun.glass.ui.win.WinApplication._runLoop(Native Method)
at javafx.graphics/com.sun.glass.ui.win.WinApplication.lambda$runLoop$3(WinApplication.java:174)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: javax.json.stream.JsonParsingException: Unexpected char 65.533 at (line no=1, column no=1, offset=0)
at org.glassfish.json.JsonTokenizer.unexpectedChar(JsonTokenizer.java:601)
at org.glassfish.json.JsonTokenizer.nextToken(JsonTokenizer.java:418)
at org.glassfish.json.JsonParserImpl$NoneContext.getNextEvent(JsonParserImpl.java:413)
at org.glassfish.json.JsonParserImpl.next(JsonParserImpl.java:363)
at org.glassfish.json.JsonReaderImpl.read(JsonReaderImpl.java:90)
at sudachi@0.4.0/com.worksap.nlp.sudachi.Settings.parseSettings(Settings.java:106)
... 57 more

[JhonnySalles]

It was my mistake of configuring it wrong.
There needs to be json files that can be found in this project in the resource folder.

and run the code below

// insert in the function you want
		try {
				
		String settings_path = App.class.getClassLoader().getResource("sudachi_fulldict.json").getPath();
		FileInputStream input = new FileInputStream(settings_path);
			
	        String settings = readAll(input);
	        Dictionary dict = new DictionaryFactory().create("src/main/resources", settings);
	        teste = dict.create();
	        SplitMode mode = SplitMode.C;
	        String text = "自然言語処理（しぜんげんごしょり、英語: natural language processing、略称：NLP）は、人間が日常的に使っている自然言語をコンピュータに処理させる一連の技術であり、人工知能と言語学の一分野である。";

	        long startTime = System.currentTimeMillis();
	        for (int i = 0; i < 1000; i++) {
	            for (Morpheme m : teste.tokenize(mode, text)) {
	                //System.out.println(m.surface());
	            }
	        }
	        long endTime = System.currentTimeMillis();

	        System.out.println("start time: " + startTime);
	        System.out.println("end time: " + endTime);
	        System.out.println("elapsed time: " + (endTime - startTime) + " ms");
	        return;
			} catch (FileNotFoundException e) {
				// TODO Auto-generated catch block
				e.printStackTrace();
			} catch (IOException e) {
				// TODO Auto-generated catch block
				e.printStackTrace();
			}

static String readAll(InputStream input) throws IOException {
        InputStreamReader isReader = new InputStreamReader(input, StandardCharsets.UTF_8);
        BufferedReader reader = new BufferedReader(isReader);
        StringBuilder sb = new StringBuilder();
        while (true) {
            String line = reader.readLine();
            if (line == null) {
                break;
            }
            sb.append(line);
        }
        return sb.toString();
    }

[kazuma-t]

CMD.EXE does not consider ' to be an escape sequence.

java.exe -jar sudachi-0.4.0.jar -s "{\"systemDict\":\"system_small.dic\"}"

Maybe this will work.