Fix: Check Before Deleting File in Workflow

[SethiShreya]

Terms

• I have searched open and closed feature requests
• I agree to follow Scribe-Data's Code of Conduct

Description

Currently, the workflow deletes the file first and then extracts the data. This approach causes an issue where if an error occurs during data extraction, the previous file is already deleted and no new data is added, resulting in data loss.

Steps to Reproduce:

• Run the following command:
  scribe-data get --lang german -dt nouns

• Carefully observe the workflow:
  a. The file is deleted first.
  b. Then the data extraction process begins.
  c. If any error occurs during the data extraction, no new data is added, and the previous file is already deleted.

Expected Behavior: The system should first verify whether the data extraction process is successful before deleting the file. If the data extraction is error-free, then the file should be deleted.

Proposed Fix:

• Add a checking mechanism before deleting the file.
• Only proceed with file deletion if the data extraction process completes successfully without errors.

Impact: This will prevent unnecessary data loss in case of errors during the extraction phase.

Contribution

No response

[andrewtavis]

Thanks for the issue, @SethiShreya!

[andrewtavis]

@SethiShreya: Do you think that including totals lexemes regardless of data type in the total output makes sense? maybe as the first row of the table? Let me know and I can make an issue for it :)

[Otom-obhazi]

can i help??

[harikrishnatp]

Hey, is this issue still active to be worked on ?

[andrewtavis]

Hey @harikrishnatp 👋 There definitely are some cases where a file is being deleted without proper input. The SQLite conversion is one case where the old file is just replaced, so you could definitely work on this a bit. Let me know if you'd have interest and I can assign/provide more info if needed :)

[harikrishnatp]

I'm very new to the codebase, but I’ve had some experience with Scribe Data before, which is why I’m interested in contributing to this project. Could you point me to the relevant parts of the codebase that would help me better understand the issue and the SQLite conversion logic? Any specific areas or starting points I should focus on would be really helpful!

[andrewtavis]

Let me get the most recent release out this evening and then I'll come back to you with suggestions, @harikrishnatp :)

[harikrishnatp]

Thank you so much :)

[andrewtavis]

Ok so the new version just got released, @harikrishnatp :) There are a few things to consider here in terms of checking before deleting a file:

• The get functionality (scribe-data get [arguments])
• The convert functionality (scribe-data convert [arguments])

I think that so far the get functionality is working well here as that's what @SethiShreya was focussed on 🎉 Would be good to test this, but then convert can be what we focus on at first. You can see the code for this command in Scribe-Data/src/scribe_data/cli/convert.py. There are lots of overwrite arguments in the various functions, but then specifically for the sqlite conversion it's not respecting when we already have a database that might be overwritten.

You can see this with:

• scribe-data get -lang German -dt prepositions # relatively small dataset to test with
• scribe-data convert -lang German -dt prepositions -if scribe_data_json_export/German/prepositions.json -ot sqlite
• Then run it again and we won't be asked if the database should be deleted
  • Beyond this, if the database has other tables in it, these are also deleted

Do you want to do some exploring and we can go from there? I'd suggest maybe bringing down German nouns and prepositions as JSON files, and then from there try converting them into csv and tsv as well as sqlite. Once the conversion is done, see what happens when you run it again with a file that will be overwritten :)

Beyond this, there is something that we could also fix for get that came up in testing for the above:

» scribe-data get -lang German -dt prepositions
Updating data for language(s): German; data type(s): Prepositions
Data updated:   0%|                                                                                                                      | 0/1 [00:00<?, ?process/s]
Existing file(s) found for German prepositions in the scribe_data_json_export directory:

1. prepositions.json

Choose an option:
1. Overwrite existing data (press 'o')
2. Skip process (press anything else)
Enter your choice: # I pressed enter
Skipping update for German prepositions.
Data updated:   0%|                                                                                                                      | 0/1 [00:02<?, ?process/s]
Updated data was saved in: MY_BASE_PATH/scribe-org/Scribe-Data/scribe_data_json_export.

See that we still get Updated data was saved in... even though we've skipped it? This might be a great first PR, and then we can move to the convert work?

Let me know what you think!