New tool to suppress leading and trailing spaces. #783

SNoiraud · 2019-02-03T17:49:49Z

This tool is looking for place names with leading or/and trailing spaces.
It also looks for in the first name and surname.

For each entry which contains leading or trailing spaces,
a row is added in a treeview.
You can see where the spaces are for each row as the name is underlined.

If you double click on the row, you can edit the Place or the Person.
For places, I search for comma. They are forbidden for lat/lon (related to (https://gramps-project.org/bugs/view.php?id=11114)

SNoiraud · 2019-02-03T17:54:17Z

codecov-io · 2019-02-03T18:03:10Z

Codecov Report

Merging #783 into master will decrease coverage by 0.05%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master     #783      +/-   ##
==========================================
- Coverage   42.05%      42%   -0.06%     
==========================================
  Files        1065     1066       +1     
  Lines      144427   144625     +198     
==========================================
  Hits        60743    60743              
- Misses      83684    83882     +198

Impacted Files	Coverage Δ
gramps/plugins/tool/removespaces.py	`0% <0%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6bca408...7712449. Read the comment docs.

Nick-Hall · 2019-02-03T19:03:21Z

Do you plan to keep the handle column?

SNoiraud · 2019-02-03T19:34:24Z

I could hide it. I need it to call the editor.

prculley · 2019-02-05T15:02:40Z

If you want to get this out to users anytime soon, you might consider packaging it as an addon. The way thing are going, I don't think we will see Gramps5.1 anytime this year.

Nick-Hall · 2019-02-05T22:52:13Z

The way thing are going, I don't think we will see Gramps5.1 anytime this year.

We will probably release v5.1 in July or August.

lordemannd · 2019-02-25T20:40:29Z

Can it also look at Alternate Names, Longitude, Latitude? (really any user enter-able text field)

lordemannd · 2019-04-09T15:39:12Z

I added the newest changes to the 5.0.1 version on windows and get the error reporting wizard.
Is this a GTK version issue? Should I just wait for 5.1?

8609344: ERROR: tool.py: line 257: Failed to start tool.
Traceback (most recent call last):
File "C:\Program Files\GrampsAIO64-5.0.1-WORK\gramps\gui\plug\tool.py", line 253, in gui_tool
callback = callback)
File "C:\Program Files\GrampsAIO64-5.0.1-WORK\gramps\plugins\tool\removespaces.py", line 77, in init
top_dialog = Glade()
File "C:\Program Files\GrampsAIO64-5.0.1-WORK\gramps\gui\glade.py", line 158, in init
self.add_from_string(data)
File "AIO/gi/overrides/Gtk.py", line 464, in add_from_string
GLib.GError: gtk-builder-error-quark: .:125:58 Invalid property: GtkScrolledWindow.max_content_height (11)

Python 3.6.4 (Success version 3.2.0 or greater installed.)
GTK+ 3.18.9 (Success version 3.10.0 or greater installed.)
PyGObject 3.26.1 (Success version 3.12.0 or greater installed.)

Nick-Hall · 2019-04-09T16:20:24Z

The max_content_height property requires Gtk 3.22 and above. We decided on Gtk v3.12 for Gramps v5.1.

lordemannd · 2019-04-10T13:33:20Z

After "Come back to Gtk 3.12 : remove max_content_height"
I get the following error. (I have checked and it looks like I have all the changes)

279416: ERROR: tool.py: line 257: Failed to start tool.
Traceback (most recent call last):
File "C:\Program Files\GrampsAIO64-5.0.1-WORK\gramps\gui\plug\tool.py", line 253, in gui_tool
callback = callback)
File "C:\Program Files\GrampsAIO64-5.0.1-WORK\gramps\plugins\tool\removespaces.py", line 77, in init
top_dialog = Glade()
File "C:\Program Files\GrampsAIO64-5.0.1-WORK\gramps\gui\glade.py", line 158, in init
self.add_from_string(data)
File "AIO/gi/overrides/Gtk.py", line 464, in add_from_string
GLib.GError: gtk-builder-error-quark: .:136:67 Invalid property: GtkScrolledWindow.propagate_natural_width (11)

SNoiraud · 2019-04-10T16:07:52Z

It's difficult to know all properties glade uses. I am on Ubuntu with Gtk 3.22.
I hope I forgot nothing else.

lordemannd · 2019-04-10T16:36:23Z

Thanks, its working now. I will do some testing.

lordemannd · 2019-04-12T03:04:01Z

It doesn't seem to be checking the alternate surname or given names.

SNoiraud · 2019-04-12T06:24:06Z

Can you attach a database example (.gramps) for that.

lordemannd · 2019-04-12T15:01:05Z

Using current Gramps, the process of backup (.gramp) and then import fixes many of these issues.
I tried using other export methods, they also result in the data being corrected.

The specific issue in my test was caused originally by cutting/pasting names from web sources. When doing this, sometimes the copied text includes control characters (carriage return, line feed, tab, vtab, bell, etc.) that are considered "white space".

In older versions of Gramps these control characters where not displayed, making it nearly impossible to find them. (but causing problems in many generated reports )
Current Gramps makes them visible.

On Windows, I can recreate the problem in alternate names easily.
(my apology for the painful detail, just trying to be clear)

in Gramps, add a new name to a person to open the name editor.
Windows, open the notepad application
add 2 lines of text.
move cursor to the end of the 1st line
left click and hold button
drag cursor to end of 2nd line
release button
you should now have the 2nd line of text highlighted (including the invisible end of the 1st line)
right click and choose copy
return to Gramps and paste the text into the Given name and Surname field.
click OK button to save this alternate name.
run the space removal tool.
notice that this alternate name is not found.
now make the alternate name the default name
rerun the tool
notice that as the default name, the issue is detected.

SNoiraud · 2019-04-13T08:32:15Z

run the space removal tool.
notice that this alternate name is not found.

It works for me. We have no alternate name here. The Given and Surname names are displayed.

now make the alternate name the default name

It works.

I attached the .gramps file for which it works. I manually edit to add the new line.
when I import it, this new line is removed only for the Given name.

So, my question now is:
Why given names are processed differently than surname or alternate names?

Perhaps solving this kind of problem would be sufficient and make this tool obsolete.

remove .txt in the file name before import it.
PR783.gramps.txt

lordemannd · 2019-04-13T10:49:12Z

In your PR783.gramps.txt it looks like the "newline" (\n) is in the middle of the name text. Did you try the "newline" character as the 1st or last character in the name?

SNoiraud · 2019-04-13T11:12:00Z

Did you try the "newline" character as the 1st or last character in the name?

Yes. in the provided file, given, surname and alternate begin with a new line.

SNoiraud · 2019-04-13T11:32:01Z

I have a patch to suppress all invisible characters in the following fields:
call_name, nick_name, family_nick_name, first_name, suffix, title, surname and prefix

I think this is a better solution

try_1.patch.txt

with this patch, this tool is obsolete for the Person.

SNoiraud · 2019-04-13T12:22:21Z

The best is perhaps to modify the MonitoredEntry class.

SNoiraud · 2019-04-13T14:00:14Z

This is the new patch for the MonitoredEntry.
It is now impossible to enter
line feed, carriage return, tab, ... in a field

try_2.patch.txt

Nick-Hall · 2019-04-13T15:40:10Z

This seems to be progressing well.

SNoiraud · 2019-04-13T16:33:35Z

I think we do not need this tool if we modify the MonitoredEntry with the try_2 patch.
It works on all views: people, events, places, media, sources and repositories

I close this PR. I'll create a new one. Can it be a bug for gramps 5.0 ?

lordemannd · 2019-04-13T17:50:33Z

it can be a bug if the we think the Gramps specification says: leading and trailing white space are not allowed in field text and will be removed.

also saying that white space is any non-printing character to include space, nl, cr, tab, vtab, bell, etc.
(i personally like this and would further say that all white space strings within the text should be replaced with a single space)

Does this apply directly for text blocks like in a Note? or should blocks of text have different rules?

BUT i think this tool is still needed:
1 - in case there are bugs in the bug fix
2 - for users that are upgrading from a previous version (like 3.4 to 5.0), they will need to be able to clean up things that older Gramps versions allowed. The upgrade I refer to is where an existing BSD DB is converted in place instead of the backup and import method.

Nick-Hall · 2019-04-13T17:53:33Z

Can it be a bug for gramps 5.0 ?

Yes, I think that allowing control characters to be pasted into an entry field could be considered a bug. Please open another PR.

We probably should allow a non-breaking space because it is used to indicate that multi-part names in Arabic and is used by the Given Names Cloud gramplet. However, you only seem to be removing control codes in the ASCII range 0-31.

Nick-Hall · 2019-04-13T17:55:04Z

Does this apply directly for text blocks like in a Note?

Notes are handled differently and may contain whitespace.

lordemannd · 2019-04-13T18:52:08Z

for Arabic, is the non-breaking space ever the 1st or last character in the field?

SNoiraud · 2019-04-15T06:25:28Z

Please open another PR.

See #811 for gramps 5.0

Nick-Hall · 2019-05-12T06:17:25Z

@SNoiraud What was the decision about this? Do we still need the tool?

SNoiraud · 2019-05-12T06:35:54Z

What was the decision about this? Do we still need the tool?

I think we need this tool after an import.
Perhaps the problem is without the name: this plugin also replace commas and not only leading and trailing spaces.
What could be the name? (standardization of input fields ?)

Nick-Hall · 2019-05-12T14:06:49Z

What could be the name? (standardization of input fields ?)

I suppose it is a form of data cleansing, so something like "Clean input data"?

Nick-Hall · 2019-05-14T12:46:43Z

@SNoiraud Is this ready to merge?

SNoiraud · 2019-05-14T17:02:45Z

We can merge it.

This tool is looking for place names with leading or/and trailing spaces. It also looks for in the first name and surname. For each entry which contains leading or trailing spaces, a row is added in a treeview. You can see where the spaces are for each row as the name is underlined. If you double click on the row, you can edit the Place or the Person. Fixes #10824

sam-m888 · 2020-01-21T06:04:43Z

Note to self [ ] the tool link to the user manual needs to be updated from https://gramps-project.org/wiki/index.php?title=Gramps_5.1_Wiki_Manual_-_Tools#Remove_leading_and_trailing_spaces to https://gramps-project.org/wiki/index.php?title=Gramps_5.1_Wiki_Manual_-_Tools# Clean_input_data You access it from the menu "Tools > Utilities > Clean input data..." See: https://sourceforge.net/p/gramps/mailman/message/36905316/

SNoiraud added the enhancement label Feb 3, 2019

SNoiraud added this to the v5.1 milestone Feb 3, 2019

sam-m888 added the Documentation label Feb 14, 2019

PQYPLZXHGF mentioned this pull request Apr 10, 2019

[GraphView] Add search capability[gramps50] gramps-project/addons-source#195

Closed

SNoiraud mentioned this pull request Apr 23, 2019

Avoid invalid characters in the entry fields #811

Merged

Nick-Hall force-pushed the FR10824 branch from e33132f to 7712449 Compare May 14, 2019 18:10

Nick-Hall merged commit 7712449 into master May 14, 2019

Nick-Hall deleted the FR10824 branch May 14, 2019 18:25

vantu5z mentioned this pull request May 30, 2019

[GraphView] Add search capability[gramps51] gramps-project/addons-source#202

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New tool to suppress leading and trailing spaces. #783

New tool to suppress leading and trailing spaces. #783

SNoiraud commented Feb 3, 2019 •

edited

SNoiraud commented Feb 3, 2019

codecov-io commented Feb 3, 2019 •

edited

Nick-Hall commented Feb 3, 2019

SNoiraud commented Feb 3, 2019

prculley commented Feb 5, 2019

Nick-Hall commented Feb 5, 2019

lordemannd commented Feb 25, 2019

lordemannd commented Apr 9, 2019

Nick-Hall commented Apr 9, 2019

lordemannd commented Apr 10, 2019 •

edited

SNoiraud commented Apr 10, 2019

lordemannd commented Apr 10, 2019

lordemannd commented Apr 12, 2019

SNoiraud commented Apr 12, 2019

lordemannd commented Apr 12, 2019 •

edited

SNoiraud commented Apr 13, 2019 •

edited

lordemannd commented Apr 13, 2019 •

edited

SNoiraud commented Apr 13, 2019 •

edited

SNoiraud commented Apr 13, 2019 •

edited

SNoiraud commented Apr 13, 2019

SNoiraud commented Apr 13, 2019

Nick-Hall commented Apr 13, 2019

SNoiraud commented Apr 13, 2019

lordemannd commented Apr 13, 2019

Nick-Hall commented Apr 13, 2019

Nick-Hall commented Apr 13, 2019

lordemannd commented Apr 13, 2019

SNoiraud commented Apr 15, 2019 •

edited

Nick-Hall commented May 12, 2019

SNoiraud commented May 12, 2019

Nick-Hall commented May 12, 2019

Nick-Hall commented May 14, 2019

SNoiraud commented May 14, 2019

sam-m888 commented Jan 21, 2020 via email

New tool to suppress leading and trailing spaces. #783

New tool to suppress leading and trailing spaces. #783

Conversation

SNoiraud commented Feb 3, 2019 • edited

SNoiraud commented Feb 3, 2019

codecov-io commented Feb 3, 2019 • edited

Codecov Report

Nick-Hall commented Feb 3, 2019

SNoiraud commented Feb 3, 2019

prculley commented Feb 5, 2019

Nick-Hall commented Feb 5, 2019

lordemannd commented Feb 25, 2019

lordemannd commented Apr 9, 2019

Nick-Hall commented Apr 9, 2019

lordemannd commented Apr 10, 2019 • edited

SNoiraud commented Apr 10, 2019

lordemannd commented Apr 10, 2019

lordemannd commented Apr 12, 2019

SNoiraud commented Apr 12, 2019

lordemannd commented Apr 12, 2019 • edited

SNoiraud commented Apr 13, 2019 • edited

lordemannd commented Apr 13, 2019 • edited

SNoiraud commented Apr 13, 2019 • edited

SNoiraud commented Apr 13, 2019 • edited

SNoiraud commented Apr 13, 2019

SNoiraud commented Apr 13, 2019

Nick-Hall commented Apr 13, 2019

SNoiraud commented Apr 13, 2019

lordemannd commented Apr 13, 2019

Nick-Hall commented Apr 13, 2019

Nick-Hall commented Apr 13, 2019

lordemannd commented Apr 13, 2019

SNoiraud commented Apr 15, 2019 • edited

Nick-Hall commented May 12, 2019

SNoiraud commented May 12, 2019

Nick-Hall commented May 12, 2019

Nick-Hall commented May 14, 2019

SNoiraud commented May 14, 2019

sam-m888 commented Jan 21, 2020 via email

SNoiraud commented Feb 3, 2019 •

edited

codecov-io commented Feb 3, 2019 •

edited

lordemannd commented Apr 10, 2019 •

edited

lordemannd commented Apr 12, 2019 •

edited

SNoiraud commented Apr 13, 2019 •

edited

lordemannd commented Apr 13, 2019 •

edited

SNoiraud commented Apr 13, 2019 •

edited

SNoiraud commented Apr 13, 2019 •

edited

SNoiraud commented Apr 15, 2019 •

edited