Add POT generation feature in Editor #39415

SkyLucilfer · 2020-06-09T13:22:37Z

Context: i18n, localization

Added "POT generation" tab under "Localization" to generate translation template files (known as POT files) from selected files.

Behind the scene, the system collects all translatable strings from the included files, and write the collected strings to a user-selected POT file.

==============
Update 23/06/2020
Added plugin support. Users can now define their custom parser to extract translation strings from custom files (.csv, .json etc.) into the POT file.

Test project:
test_project.zip

akien-mga

Only reviewed the first commit so far, but here are some initial comments.
Great work overall!

core/io/pot_generator.cpp

core/io/pot_generator.h

editor/project_settings_editor.cpp

akien-mga · 2020-06-17T14:21:24Z

I got a crash when trying to write the POT file using the files from the ZIP (I initialized a project with touch project.godot and opened it directly). Note that I rebased your branch on the current master before compiling, so it might have an impact.

This was the first time opening the project, and I added the two .gd files and the .tscn to the POT generation tab.

$ godot-git -e
Godot Engine v4.0.dev.custom_build.6a73c0eb9 - https://godotengine.org
DisplayServer::_create_window 0 want rect: 448, 240, 1024, 600 got rect 0, 0, 1920, 1051
DisplayServer::_window_changed: 0 rect: 1920, 29, 1024, 600
DisplayServer::_window_changed: 0 rect: 1920, 29, 1920, 1051
 
screenpos: 64, 8
WARNING: Property not found: locale/translations_pot_files
     at: _get (core/project_settings.cpp:197)
WARNING: Property not found: locale/translations_pot_files
     at: _get (core/project_settings.cpp:197)
ERROR: Error calling method from signal '_settings_changed': 'ProjectSettingsEditor::_settings_changed': Method not found.
   at: _process_operation_list (core/undo_redo.cpp:291)
ERROR: Texture (binding: 1, index 0) is not a valid texture.
   at: uniform_set_create (drivers/vulkan/rendering_device_vulkan.cpp:4543)
ERROR: Texture (binding: 1, index 0) is not a valid texture.
   at: uniform_set_create (drivers/vulkan/rendering_device_vulkan.cpp:4543)
ERROR: Error calling method from signal '_settings_changed': 'ProjectSettingsEditor::_settings_changed': Method not found.
   at: _process_operation_list (core/undo_redo.cpp:291)
ERROR: Texture (binding: 1, index 0) is not a valid texture.
   at: uniform_set_create (drivers/vulkan/rendering_device_vulkan.cpp:4543)
ERROR: Texture (binding: 1, index 0) is not a valid texture.
   at: uniform_set_create (drivers/vulkan/rendering_device_vulkan.cpp:4543)
ERROR: Error calling method from signal '_settings_changed': 'ProjectSettingsEditor::_settings_changed': Method not found.
   at: _process_operation_list (core/undo_redo.cpp:291)
ERROR: Condition "err" is true. Returning: err
   at: load_source_code (modules/gdscript/gdscript.cpp:840)
ERROR: Cannot load source code from file 'res://simpletest.gd'.
   at: load (modules/gdscript/gdscript.cpp:2257)
ERROR: Failed loading resource: res://simpletest.gd.
   at: _load (core/io/resource_loader.cpp:198)
ERROR: res://Node3D.tscn:29 - Parse Error: [ext_resource] referenced non-loaded resource at: res://simpletest.gd
   at: _parse_ext_resource (scene/resources/resource_format_text.cpp:183)
ERROR: res://Node3D.tscn:29 - Parse Error: [ext_resource] referenced non-loaded resource at: res://simpletest.gd
   at: _parse_node_tag (scene/resources/resource_format_text.cpp:282)
ERROR: Failed loading resource: res://Node3D.tscn.
   at: _load (core/io/resource_loader.cpp:198)
handle_crash: Program crashed with signal 11
Dumping the backtrace. Please include this when reporting the bug on https://github.com/godotengine/godot/issues
[1] /lib64/libc.so.6(+0x3a3f0) [0x7f21be2b03f0] (??:0)
[2] Ref<SceneState>::ref(Ref<SceneState> const&) (/home/akien/Projects/godot/godot.git/./core/reference.h:62)
[3] Ref<SceneState>::Ref(Ref<SceneState> const&) (/home/akien/Projects/godot/godot.git/./core/reference.h:179)
[4] PackedScene::get_state() (/home/akien/Projects/godot/godot.git/scene/resources/packed_scene.cpp:1669)
[5] POTGenerator::generate_pot(String const&) (/home/akien/Projects/godot/godot.git/core/io/pot_generator.cpp:67 (discriminator 1))
[6] ProjectSettingsEditor::_translation_pot_generate(String const&) (/home/akien/Projects/godot/godot.git/editor/project_settings_editor.cpp:1580)
[7] void call_with_variant_args_helper<ProjectSettingsEditor, String const&, 0ul>(ProjectSettingsEditor*, void (ProjectSettingsEditor::*)(String const&), Variant const**, Callable::CallError&, IndexSequence<0ul>) (/home/akien/Projects/godot/godot.git/./core/callable_method_pointer.h:132 (discriminator 6))
[8] void call_with_variant_args<ProjectSettingsEditor, String const&>(ProjectSettingsEditor*, void (ProjectSettingsEditor::*)(String const&), Variant const**, int, Callable::CallError&) (/home/akien/Projects/godot/godot.git/./core/callable_method_pointer.h:158)
[9] CallableCustomMethodPointer<ProjectSettingsEditor, String const&>::call(Variant const**, int, Variant&, Callable::CallError&) const (/home/akien/Projects/godot/godot.git/./core/callable_method_pointer.h:185)
[10] Callable::call(Variant const**, int, Variant&, Callable::CallError&) const (/home/akien/Projects/godot/godot.git/core/callable.cpp:54)
[11] Object::emit_signal(StringName const&, Variant const**, int) (/home/akien/Projects/godot/godot.git/core/object.cpp:1170)
[12] Object::emit_signal(StringName const&, Variant const&, Variant const&, Variant const&, Variant const&, Variant const&) (/home/akien/Projects/godot/godot.git/core/object.cpp:1225)
[13] EditorFileDialog::_action_pressed() (/home/akien/Projects/godot/godot.git/editor/editor_file_dialog.cpp:444 (discriminator 3))
[14] EditorFileDialog::_file_entered(String const&) (/home/akien/Projects/godot/godot.git/editor/editor_file_dialog.cpp:229)
[15] void call_with_variant_args_helper<EditorFileDialog, String const&, 0ul>(EditorFileDialog*, void (EditorFileDialog::*)(String const&), Variant const**, Callable::CallError&, IndexSequence<0ul>) (/home/akien/Projects/godot/godot.git/./core/callable_method_pointer.h:132 (discriminator 6))
[16] void call_with_variant_args<EditorFileDialog, String const&>(EditorFileDialog*, void (EditorFileDialog::*)(String const&), Variant const**, int, Callable::CallError&) (/home/akien/Projects/godot/godot.git/./core/callable_method_pointer.h:158)
[17] CallableCustomMethodPointer<EditorFileDialog, String const&>::call(Variant const**, int, Variant&, Callable::CallError&) const (/home/akien/Projects/godot/godot.git/./core/callable_method_pointer.h:185)
[18] Callable::call(Variant const**, int, Variant&, Callable::CallError&) const (/home/akien/Projects/godot/godot.git/core/callable.cpp:54)
[19] Object::emit_signal(StringName const&, Variant const**, int) (/home/akien/Projects/godot/godot.git/core/object.cpp:1170)
[20] Object::emit_signal(StringName const&, Variant const&, Variant const&, Variant const&, Variant const&, Variant const&) (/home/akien/Projects/godot/godot.git/core/object.cpp:1225)
[21] LineEdit::_gui_input(Ref<InputEvent>) (/home/akien/Projects/godot/godot.git/scene/gui/line_edit.cpp:295 (discriminator 3))
[22] MethodBind1<Ref<InputEvent> >::call(Object*, Variant const**, int, Callable::CallError&) (/home/akien/Projects/godot/godot.git/./core/method_bind.gen.inc:775 (discriminator 12))
[23] Object::call_multilevel(StringName const&, Variant const**, int) (/home/akien/Projects/godot/godot.git/core/object.cpp:737 (discriminator 1))
[24] Object::call_multilevel(StringName const&, Variant const&, Variant const&, Variant const&, Variant const&, Variant const&) (/home/akien/Projects/godot/godot.git/core/object.cpp:836)
[25] Viewport::_gui_input_event(Ref<InputEvent>) (/home/akien/Projects/godot/godot.git/scene/main/viewport.cpp:2305 (discriminator 1))
[26] Viewport::input(Ref<InputEvent> const&, bool) (/home/akien/Projects/godot/godot.git/scene/main/viewport.cpp:2921 (discriminator 3))
[27] Window::_window_input(Ref<InputEvent> const&) (/home/akien/Projects/godot/godot.git/scene/main/window.cpp:886 (discriminator 1))
[28] Viewport::_sub_windows_forward_input(Ref<InputEvent> const&) (/home/akien/Projects/godot/godot.git/scene/main/viewport.cpp:2888)
[29] Viewport::input(Ref<InputEvent> const&, bool) (/home/akien/Projects/godot/godot.git/scene/main/viewport.cpp:2911 (discriminator 1))
[30] Window::_window_input(Ref<InputEvent> const&) (/home/akien/Projects/godot/godot.git/scene/main/window.cpp:886 (discriminator 1))
[31] void call_with_variant_args_helper<Window, Ref<InputEvent> const&, 0ul>(Window*, void (Window::*)(Ref<InputEvent> const&), Variant const**, Callable::CallError&, IndexSequence<0ul>) (/home/akien/Projects/godot/godot.git/./core/callable_method_pointer.h:132 (discriminator 6))
[32] void call_with_variant_args<Window, Ref<InputEvent> const&>(Window*, void (Window::*)(Ref<InputEvent> const&), Variant const**, int, Callable::CallError&) (/home/akien/Projects/godot/godot.git/./core/callable_method_pointer.h:158)
[33] CallableCustomMethodPointer<Window, Ref<InputEvent> const&>::call(Variant const**, int, Variant&, Callable::CallError&) const (/home/akien/Projects/godot/godot.git/./core/callable_method_pointer.h:185)
[34] Callable::call(Variant const**, int, Variant&, Callable::CallError&) const (/home/akien/Projects/godot/godot.git/core/callable.cpp:54)
[35] DisplayServerX11::_dispatch_input_event(Ref<InputEvent> const&) (/home/akien/Projects/godot/godot.git/platform/linuxbsd/display_server_x11.cpp:2371)
[36] DisplayServerX11::_dispatch_input_events(Ref<InputEvent> const&) (/home/akien/Projects/godot/godot.git/platform/linuxbsd/display_server_x11.cpp:2355)
[37] Input::_parse_input_event_impl(Ref<InputEvent> const&, bool) (/home/akien/Projects/godot/godot.git/core/input/input.cpp:590)
[38] Input::parse_input_event(Ref<InputEvent> const&) (/home/akien/Projects/godot/godot.git/core/input/input.cpp:440)
[39] Input::flush_accumulated_events() (/home/akien/Projects/godot/godot.git/core/input/input.cpp:830)
[40] DisplayServerX11::process_events() (/home/akien/Projects/godot/godot.git/./core/os/mutex.h:70)
[41] OS_LinuxBSD::run() (/home/akien/Projects/godot/godot.git/platform/linuxbsd/os_linuxbsd.cpp:239)
[42] godot-git(main+0xfa) [0x19eba5c] (/home/akien/Projects/godot/godot.git/platform/linuxbsd/godot_linuxbsd.cpp:57)
[43] /lib64/libc.so.6(__libc_start_main+0xea) [0x7f21be29ccda] (??:0)
[44] godot-git(_start+0x2a) [0x19eb8ba] (??:?)
-- END OF BACKTRACE --
Aborted (core dumped)

SkyLucilfer · 2020-06-18T11:27:02Z

@akien-mga I've looked at the crash. The .tscn file that I've included have dependencies on other resources in the project, but I did not put them together in the ZIP. Hence the .tscn is corrupted.
For a scene created from a project itself, should be ok.

akien-mga · 2020-06-18T11:30:38Z

I've looked at the crash. The .tscn file that I've included have dependencies on other resources in the project, but I did not put them together in the ZIP. Hence the .tscn is corrupted.

Alright. That's a good testcase still, the engine shouldn't crash even if given invalid input, so you should use ERR_* checks where relevant to avoid crashing.

For example:

translation_strings = _parse_scene(res->get_state());

This will crash if res->get_state() is invalid, as there's no validation done in _parse_scene. Either _parse_scene should validate the reference and abort, or it should be done before calling _parse_scene.

akien-mga

Looks great!

editor/editor_translation_parser.cpp

editor/editor_translation_parser.h

editor/pot_generator.cpp

modules/gdscript/editor/gdscript_translation_parser_plugin.cpp

modules/gdscript/editor/gdscript_translation_parser_plugin.h

editor/editor_translation_parser.cpp

akien-mga · 2020-07-01T07:42:02Z

Tested locally and re-reviewed the code, amazing work! It's good to merge IMO.

One last change though, could you generate and fill the documentation for the new class and methods? See https://docs.godotengine.org/en/latest/community/contributing/updating_the_class_reference.html, the TL;DR is godot --doctool . and fill the added <description>s.

doc/classes/EditorPlugin.xml

doc/classes/EditorTranslationParserPlugin.xml

akien-mga · 2020-07-02T20:22:50Z

Thanks!

Calinou added enhancement topic:core topic:editor labels Jun 9, 2020

Calinou added this to the 4.0 milestone Jun 9, 2020

SkyLucilfer force-pushed the PotGeneration branch 3 times, most recently from 3c470b6 to c2b15c8 Compare June 11, 2020 13:00

SkyLucilfer changed the title ~~Add functionality to extract translation strings~~ Add POT generation feature in Editor Jun 11, 2020

akien-mga reviewed Jun 17, 2020

View reviewed changes

SkyLucilfer force-pushed the PotGeneration branch 4 times, most recently from 0fd367d to a07bd25 Compare June 23, 2020 12:21

SkyLucilfer requested review from bojidar-bg and vnen as code owners June 23, 2020 12:21

SkyLucilfer force-pushed the PotGeneration branch from a07bd25 to 71beb38 Compare June 23, 2020 12:31

akien-mga reviewed Jun 23, 2020

View reviewed changes

modules/gdscript/editor/gdscript_translation_parser_plugin.h Outdated Show resolved Hide resolved

SkyLucilfer force-pushed the PotGeneration branch 2 times, most recently from 7eeaab5 to 9a12318 Compare June 30, 2020 09:32

akien-mga reviewed Jul 1, 2020

View reviewed changes

editor/editor_translation_parser.cpp Outdated Show resolved Hide resolved

SkyLucilfer force-pushed the PotGeneration branch 2 times, most recently from 6bba3ca to ff829e7 Compare July 1, 2020 17:44

SkyLucilfer requested a review from a team as a code owner July 1, 2020 17:44

akien-mga reviewed Jul 1, 2020

View reviewed changes

doc/classes/EditorPlugin.xml Show resolved Hide resolved

SkyLucilfer force-pushed the PotGeneration branch from ff829e7 to e3f1128 Compare July 2, 2020 07:20

akien-mga approved these changes Jul 2, 2020

View reviewed changes

akien-mga reviewed Jul 2, 2020

View reviewed changes

doc/classes/EditorTranslationParserPlugin.xml Outdated Show resolved Hide resolved

SkyLucilfer added 2 commits July 2, 2020 14:47

Added "POT generation" feature under "Localization" in the Editor

1db29d0

Add translation parser plugin support

efb4609

SkyLucilfer force-pushed the PotGeneration branch from e3f1128 to efb4609 Compare July 2, 2020 12:47

akien-mga merged commit f9c2f35 into godotengine:master Jul 2, 2020

SkyLucilfer mentioned this pull request Aug 9, 2020

Update i18n gettext tutorial to include POT generation, plurals and context support godotengine/godot-docs#3884

Closed

akien-mga mentioned this pull request Nov 2, 2020

Cannot use localized resx files in exported game #43277

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add POT generation feature in Editor #39415

Add POT generation feature in Editor #39415

SkyLucilfer commented Jun 9, 2020 •

edited

Loading

akien-mga left a comment

akien-mga commented Jun 17, 2020

SkyLucilfer commented Jun 18, 2020

akien-mga commented Jun 18, 2020 •

edited

Loading

akien-mga left a comment

akien-mga commented Jul 1, 2020

akien-mga commented Jul 2, 2020

Add POT generation feature in Editor #39415

Add POT generation feature in Editor #39415

Conversation

SkyLucilfer commented Jun 9, 2020 • edited Loading

akien-mga left a comment

Choose a reason for hiding this comment

akien-mga commented Jun 17, 2020

SkyLucilfer commented Jun 18, 2020

akien-mga commented Jun 18, 2020 • edited Loading

akien-mga left a comment

Choose a reason for hiding this comment

akien-mga commented Jul 1, 2020

akien-mga commented Jul 2, 2020

SkyLucilfer commented Jun 9, 2020 •

edited

Loading

akien-mga commented Jun 18, 2020 •

edited

Loading