New file format (testing appreciated) #522

diegogangl · 2020-12-06T23:22:34Z

This is a branch for the new file format based on a spec from #279 with help from @johnnybubonic.
The spec is documented here: https://github.com/getting-things-gnome/gtg/blob/file_format_2/docs/contributors/file%20format.md

Testing super appreciated! GTG makes tons of backups, but this should still be as rock solid as possible.

closes #279

How to test:

Basically open GTG dev and play around. See if anything is saved incorrectly or there are any weird errors.
Another option is to use the import scripts from the scripts folder. You can import your tasks from GTG stable and test with them.

nekohayo · 2021-01-19T15:35:01Z

On trying to launch with my personal dataset on the file_format_2 branch on my Ubuntu GTG smoke testing machine, while having previously launched it under master, I get this traceback preventing startup (and the process hangs there, you have to kill it):

2021-01-19 10:29:22,811 - DEBUG - application:set_debug_flag:580 - Debug output enabled.
2021-01-19 10:29:22,881 - DEBUG - __init__:__init__:58 - Backends found: ['backend_localfile', 'backend_signals']
2021-01-19 10:29:23,089 - WARNING - backend_localfile:this_is_the_first_run:147 - Found old file. Running versioning code.
Traceback (most recent call last):
  File "/home/user/gtg/.local_build/install/lib/python3.8/site-packages/GTG/gtk/application.py", line 95, in do_startup
    for backend_dic in BackendFactory().get_saved_backends_list():
  File "/home/user/gtg/.local_build/install/lib/python3.8/site-packages/GTG/backends/__init__.py", line 167, in get_saved_backends_list
    dic["backend"].this_is_the_first_run(None)
  File "/home/user/gtg/.local_build/install/lib/python3.8/site-packages/GTG/backends/backend_localfile.py", line 149, in this_is_the_first_run
    tree = versioning.convert(old_path, self.datastore)
AttributeError: 'Backend' object has no attribute 'datastore'
Traceback (most recent call last):
  File "/home/user/gtg/.local_build/install/lib/python3.8/site-packages/GTG/gtk/application.py", line 122, in do_activate
    self.init_shared()
  File "/home/user/gtg/.local_build/install/lib/python3.8/site-packages/GTG/gtk/application.py", line 158, in init_shared
    self.init_browser()
  File "/home/user/gtg/.local_build/install/lib/python3.8/site-packages/GTG/gtk/application.py", line 166, in init_browser
    self.browser = MainWindow(self.req, self)
  File "/home/user/gtg/.local_build/install/lib/python3.8/site-packages/GTG/gtk/browser/main_window.py", line 69, in __init__
    self.config = self.req.get_config('browser')
AttributeError: 'NoneType' object has no attribute 'get_config'

diegogangl · 2021-01-24T03:11:49Z

Can't replicate. Maybe it's a race condition (register_datastore() hasn't been called yet)?

Neui · 2021-01-25T01:38:05Z

I was able to reproduce. Looking at it, self.datastore is not set. This is set by GenericBackend.register_datastore which is called part of the TaskSource.__init__, which is only constructed in DataStore.register_backend, which is only called in Application.do_startup inside the loop. However, this_is_the_first_run is only called inside the BackendFactory.get_saved_backends_list, which is called before that loop body runs.

In my case, the loop before calling this_is_the_first_run (for backend in config.get_all_backends():), config.get_all_backends() returns nothing since the backends.conf config file either doesn't exists or is is empty.

I don't know how it would work for you.

diegogangl · 2021-01-25T02:06:30Z

Deleting backends.conf and gtg_data.xml (if present) triggers the bug

diegogangl · 2021-01-26T00:22:13Z

Pushed a fix for that bug

nekohayo · 2021-01-27T02:29:29Z

Now it launches, but with my personal data set (anonymized version) auto-imported/converted by your implementation:

Tags and subtasks are not shown at all inside the task editor, only the title and description are shown
The tasks/subtasks hierarchy is completely gone from the task browser's list of tasks, for any given tag
projects.xml is somehow not updated to point to the new gtg_data.xml file but still works? Is this branch doing away entirely with the notion of projects (aka profiles), which may have been interesting for the future?

diegogangl · 2021-01-27T17:18:54Z

projects.xml is somehow not updated to point to the new gtg_data.xml file but still works? Is this branch doing away entirely with the notion of projects (aka profiles), which may have been interesting for the future?

This was mentioned in #279. We don't really need another xml file since everything is inside gtg_data.xml. You can have different profiles by loading different xml files directly.

And fix some broken ones from the rebase

Following the spec and XML data types

For translations

Sometimes the first run task can be called before initialization. If versioning isn't called then, it can create a new empty file instead.

This avoids a freeze when backends.conf is empty or doesn't exist. We need to call this_is_first_run() *after* the datastore has been set in the backend

We moved to emojis earlier than anticipated, so there's no need to keep the old icons around. fixes #540

This used to be only for @-style IDs, but we have to do it for the older IDs too. Otherwise GTG can't connect subtasks to their parents (since they are using UUIDs)

CDATA takes about everything but that combination of characters, so we need to encode them.

fixes #557

tkdchen · 2021-02-21T09:51:09Z

Synced the code and the latest one on master branch is fa7b34a.

After ./launch.sh, the gtg dialog is empty. My tasks are still available in tmp/default/xdg/data/gtg/gtg_tasks.xml.

Do I have to run scripts/import_my_tasks_into_debug_tasks.sh to import the data into the new format?

I gave a try to run the script and got:

Running the development/debug version - using separate user directories
Your data is in the 'tmp' subdirectory with the 'default' dataset.
-----------------------------------------------------------------------
cp: cannot stat '/home/chen/.local/share/gtg/*': No such file or directory
cp: cannot stat '/home/chen/.config/gtg/*': No such file or directory

What prerequisite is required to run the script?

morhe-blip · 2021-02-21T15:26:44Z

Confirmed in #565

If you work with the dev version (as I do), the script should not work, because there is no data .local/ folder.
This folder is used for the 'regular' version of GTG.

WhilelM · 2021-02-28T13:13:29Z

Same problem that @tkdchen
Synced the code and the latest one on master branch is 4a608d2
After ./launch.sh, the gtg dialog is empty. My tasks are still available in tmp/default/xdg/data/gtg/f4cf1957-8b10-4562-917b-812e5e170377.xml

I could work with my data when checkout commit 5442868

diegogangl · 2021-02-28T21:21:08Z

@WhilelM is this data from older versions of GTG (0.3 and earlier)? Could you try renaming that file to gtg_tasks.xml?

WhilelM · 2021-03-01T15:10:23Z

@WhilelM is this data from older versions of GTG (0.3 and earlier)? Could you try renaming that file to gtg_tasks.xml?

I have backup from 2013 and I use debian, so I think so.
I tried git checkout master and copied in a new file cp tmp/default/xdg/data/gtg/f4cf1957-8b10-4562-917b-812e5e170377.xml tmp/default/xdg/data/gtg/gtg_tasks.xml. no more success =/

WhilelM · 2021-03-22T15:48:50Z

How can I convert my data to new format ?

diegogangl · 2021-03-22T16:46:37Z

HI @WhilelM, it should convert automatically. Try using master, not this branch though.

WhilelM · 2021-12-31T14:42:32Z

I still have issue with my data. I'm stucked with commit 5442868 and can't load any task if I use anything less older.
How gtg would convert it automatically ?

diegogangl · 2022-01-01T23:42:50Z

That's weird, there's no change in the file format there or in any recent commit. Are you getting any errors in the terminal?

diegogangl added enhancement priority:critical labels Dec 6, 2020

diegogangl added this to the 0.5 "You Can (Not) Improve Performance" milestone Dec 6, 2020

diegogangl force-pushed the file_format_2 branch 2 times, most recently from 9b32801 to a9f337d Compare January 31, 2021 22:36

diegogangl force-pushed the file_format_2 branch from be941e0 to 0d6999b Compare February 11, 2021 00:02

diegogangl added 17 commits February 18, 2021 20:52

Add new file format spec

ac6b43e

Add backends to configuration system

fe23e9a

Save and read backends from configuration files

051c8f9

Remove projects.xml path from dirs

8a86e74

Load tasks xml once

8c717a7

Add versioning module and initial code

be28436

Fix future warning

d0968c2

Fix paths for versioning

bca216f

Remove unused return

92f7bc2

Read and write with the new file format

56a0928

Remove warning for config

4da160f

Make sure the parser doesn't strip cdata

8a4e8ce

Fix bug when creating initial file

9a8da61

Read tags from data file

4af3531

Load tags in tasks via tag id

15e0c1b

Remove unused imports

e481e76

Save tags with new format

0bb4e04

diegogangl added 16 commits February 18, 2021 20:53

Store and read saved searches

620c517

Add missing tag to first run tasks

023d4ff

Remove unused imports

0980c11

And fix some broken ones from the rebase

Update names for test data sets

72796c8

Remove # in tags color

ea02d2b

Following the spec and XML data types

Hardcode UUIDs in first run tasks

7009e75

For translations

Run versioning code earlier

faad5cb

Sometimes the first run task can be called before initialization. If versioning isn't called then, it can create a new empty file instead.

Update tag adding for new format

6bdbd2f

Move initialization to register_backend

4626107

This avoids a freeze when backends.conf is empty or doesn't exist. We need to call this_is_first_run() *after* the datastore has been set in the backend

Remove icon versioning

122ee1b

We moved to emojis earlier than anticipated, so there's no need to keep the old icons around. fixes #540

Store all IDs to UUIDs in versioning

452d6b2

This used to be only for @-style IDs, but we have to do it for the older IDs too. Otherwise GTG can't connect subtasks to their parents (since they are using UUIDs)

Fix missing logger in localfile backend

756333b

Add tags even if task content is present

4734aaa

Guard against ]]> in content

b4f2801

CDATA takes about everything but that combination of characters, so we need to encode them.

Make sure tags have @ when adding it in content

c1854d8

fixes #557

Don't add @ to tags in the batch tagger

3aff2f8

diegogangl force-pushed the file_format_2 branch from 4f87e93 to 3aff2f8 Compare February 18, 2021 23:54

diegogangl merged commit fa7b34a into master Feb 19, 2021

morhe-blip mentioned this pull request Feb 24, 2021

No data after new_file_format #565

Closed

diegogangl deleted the file_format_2 branch March 2, 2021 21:55

nekohayo added the maintainability Automated tests suite, tooling, refactoring, or anything that makes it easier for developers label Feb 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New file format (testing appreciated) #522

New file format (testing appreciated) #522

diegogangl commented Dec 6, 2020

nekohayo commented Jan 19, 2021

diegogangl commented Jan 24, 2021

Neui commented Jan 25, 2021

diegogangl commented Jan 25, 2021

diegogangl commented Jan 26, 2021

nekohayo commented Jan 27, 2021

diegogangl commented Jan 27, 2021

tkdchen commented Feb 21, 2021

morhe-blip commented Feb 21, 2021

WhilelM commented Feb 28, 2021

diegogangl commented Feb 28, 2021

WhilelM commented Mar 1, 2021

WhilelM commented Mar 22, 2021

diegogangl commented Mar 22, 2021

WhilelM commented Dec 31, 2021

diegogangl commented Jan 1, 2022

New file format (testing appreciated) #522

New file format (testing appreciated) #522

Conversation

diegogangl commented Dec 6, 2020

How to test:

nekohayo commented Jan 19, 2021

diegogangl commented Jan 24, 2021

Neui commented Jan 25, 2021

diegogangl commented Jan 25, 2021

diegogangl commented Jan 26, 2021

nekohayo commented Jan 27, 2021

diegogangl commented Jan 27, 2021

tkdchen commented Feb 21, 2021

morhe-blip commented Feb 21, 2021

WhilelM commented Feb 28, 2021

diegogangl commented Feb 28, 2021

WhilelM commented Mar 1, 2021

WhilelM commented Mar 22, 2021

diegogangl commented Mar 22, 2021

WhilelM commented Dec 31, 2021

diegogangl commented Jan 1, 2022