Date format parsed incorrectly in MDOC files

[JuhaHuiskonen]

I am trying to read tilt information from MDOC files. However I am getting this error

1	2024-01-29 21:51:45 [INFO] /opt/pyp/bin/run/pyp:3260 | Job 20259999_1 (v0.6.0) launching on r18c31.bullx using 1 task(s) and 4 GB of RAM
2	2024-01-29 21:51:45 [INFO] /opt/pyp/bin/run/pyp:3187 | Filesystem                                      Size  Used Avail Use% Mounted on
3	2024-01-29 21:51:45 [INFO] /opt/pyp/bin/run/pyp:3187 | 10.140.15.146@o2ib:10.140.15.145@o2ib:/scratch   20T  3.2T   17T  17% /scratch/project_2007321/users/huiskone/nextPYP/tmp
4	2024-01-29 21:51:45 [INFO] /opt/pyp/bin/run/pyp:1345 | Working path: /scratch/project_2007321/users/huiskone/nextPYP/tmp/huiskone/20259999_1/lamella10_ts_003
5	2024-01-29 21:51:45 [INFO] pyp/inout/utils/core.py:17 | Retrieving .pyp_config.toml
6	2024-01-29 21:51:45 [INFO] pyp/inout/utils/core.py:17 | Retrieving tomo-preprocessing-ewNgSe0LltKQav15.micrographs
7	2024-01-29 21:51:45 [INFO] pyp/inout/utils/core.py:17 | Retrieving tomo-preprocessing-ewNgSe0LltKQav15.films
8	2024-01-29 21:51:45 [INFO] /opt/pyp/bin/run/pyp:57 | Loading results took: 00h 00m 00s
9	2024-01-29 21:51:45 [INFO] /opt/pyp/bin/run/pyp:57 | Total time in tomo_swarm 00h 00m 00s seconds
10	Traceback (most recent call last):
11	  File "/opt/pyp/src/pyp/preprocess/core.py", line 906, in frames_from_mdoc
12	    data_output = datetime.datetime.strptime(time, date_pattern)
13	  File "/usr/local/envs/pyp/lib/python3.8/_strptime.py", line 568, in _strptime_datetime
14	    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
15	  File "/usr/local/envs/pyp/lib/python3.8/_strptime.py", line 349, in _strptime
16	    raise ValueError("time data %r does not match format %r" %
17	ValueError: time data '21-Sep-2023  22:05:41' does not match format '%y-%b-%d  %H:%M:%S'
18	
19	During handling of the above exception, another exception occurred:
20	
21	Traceback (most recent call last):
22	  File "/opt/pyp/bin/run/pyp", line 3478, in <module>
23	    tomo_swarm(args.path, args.file, args.debug, args.keep, args.skip)
24	  File "/opt/pyp/src/pyp/utils/timer.py", line 78, in wrapper_timer
25	    return func(*args, **kwargs)
26	  File "/opt/pyp/bin/run/pyp", line 1429, in tomo_swarm
27	    ] = preprocess.read_tilt_series(
28	  File "/opt/pyp/src/pyp/preprocess/core.py", line 177, in read_tilt_series
29	    tilts = frames_from_mdoc(mdocs, parameters)
30	  File "/opt/pyp/src/pyp/preprocess/core.py", line 908, in frames_from_mdoc
31	    raise Exception(f"{time} cannot be matched by the pattern {date_pattern}")
32	Exception: 21-Sep-2023  22:05:41 cannot be matched by the pattern %y-%b-%d  %H:%M:%S
33	2024-01-29 21:51:45 [ERROR] /opt/pyp/bin/run/pyp:3483 | PYP (tomoswarm) failed

Looks like my time format in MDOCs is not as expected?

[tofushan]

Hi,

Yes, you are right. The time in your mdoc file has different format. We can simply add this format in the next release.

To fix it, you can:

1. Modify the time in your mdoc files to year-month-day instead of day-month-year
2. Use the pattern to detect acquisition order and tilt angles from your filenames (in this case your movie files need to contain these information). Although we do not have a documentation on how to use the pattern yet, you may find this thread Tutorial runs, but not our data #40 helpful.

Hope this helps!

[JuhaHuiskonen]

To modify the MDOCs, I am using sed -i s/21-Sep-2023/23-Sep-21/ lamella10_ts_003.mrc.mdoc This seems to result in the correct date format.

[tofushan]

This is great. Also double-check if you had collected the data in a different day other than 21-Sep-2023

[abartesaghi]

Version 0.6.1 now supports multiple date formats.

[JuhaHuiskonen]

Great, thank you!