final merge: Offline SBOL3 <-> GenBank Converter; Integration with previous online Converter #186

mohitdmak · 2022-10-19T18:36:36Z

About the Project:

⇒ Description:

Until now, conversion between the file formats of GenBank and SBOL3 used to happen using an online converter, which was called by the SBOL-utilities package.
This was not an ideal approach for the conversion mostly due to 3 reasons:
1. The online converter's hosting service could have downtime, leading to outages and unavailability of the conversion to and from GenBank and SBOL3, and moreover, users could not infer the reasons for this when it happened.
2. The online converter didn't completely convert all aspects of the GenBank file (including those properties which could not be directly stored in SBOL3, but could be useful in a round trip to restore back into GenBank). The new converter handles this really well and is able to store all properties and aspects of the GenBank which do not have a corresponding SBOL3 field.
3. The online converter used to convert GenBank files to SBOL2 and then SBOL-utilities used to locally convert the SBOL2 to SBOL3, this was not ideal too, and the new offline converter directly supports conversion between GenBank and SBOL3.
Thus, this feature addition would solve all the above issues, and support direct, offline conversion between SBOL3 and GenBank file formats.

⇒ Project Repository:
https://github.com/SynBioDex/SBOL-utilities

⇒ Project Proposal:
[google-summer-of-code-2022-proposal.pdf](https://s3-us-west-2.amazonaws.com/secure.notion-static.com/bfc495bd-2261-4700-bf4c-82fe3770af02/google-summer-of-code-2022-proposal.pdf)

⇒ Project Profile:
[Google Summer of Code](https://summerofcode.withgoogle.com/programs/2022/projects/IvKQhkk9)

⇒ Project Description:
nrnb/GoogleSummerOfCode#183

⇒ Constituent Pull Requests:
https://github.com/SynBioDex/SBOL-utilities/pulls?q=label%3Asbol3-genbank+

Project Blog:

Detailed notes about design decisions, issue priorities, future prospects pertaining to each relevant PR have been documented in my blog.
It also includes timelines and dependencies for each PR.
NOTE: This blog would be of great help to contributors who are willing to understand, in depth, the progress of the whole project from ideation to implementation. You are welcome to approach me for any feedback, query, or request pertaining to this project through the blog or through mail.
Here is the link to the blog: https://mohitdmak.notion.site/GSoC-22-Conversion-between-SBOL3-and-GenBank-0c2efaf26fde493ab7914581322458e0

Credits:

I was able to bring this project to a successful completion under the guidance of my mentors for the project:

Co-authored-by: Tom Mitchell <tcmitchell@users.noreply.github.com>

add: initial simple conversion gb/sbol3 with sample files

…ures

Co-authored-by: Tom Mitchell <tcmitchell@users.noreply.github.com>

…o gb2sbol3__1

…logging; del: tmp scripts; formatting

add: genbank conversion support for multiple records and features

… legacy converter

…/SynBioDex/SBOL-utilities into sbol3togenbank

jakebeal · 2022-10-22T20:01:44Z

I have begun testing with the iGEM distribution, and run into an error. It appears that converted GenBank files are getting a 0-based "RangeStart" value rather than 1-based, which is both incorrect and results in invalid SBOL when the range should start at 1. An example of such a file:

package.nt.txt

This was made by converting GenBank files such as the following:
111772.gb.txt

jakebeal

The 0-based range error needs to be corrected.

mohitdmak · 2022-10-23T04:39:05Z

The 0-based range error needs to be corrected.

This turned out to be an issue with BioPython, as it parsed the start Locations of GenBank files based on 0-indexing instead of 1-indexing. I have created a PR for the fix, where we convert the start value to 1-indexed when converting from GenBank to SBOL3 and 0-indexed for the opposite direction.

fix: 0-indexed parsing of start locations by BioPython bug

jakebeal · 2023-03-07T22:49:36Z

It appears that something changed between biopython 1.79 and 1.81 that is causing the tests to fail. This will need investigation.

SeqFeature is no longer hashable as of BioPython 1.80. Try an alternate implementation that avoids using SeqFeature instances as keys in a dict. Store the loc_positions on a new property of SeqFeature. Akin to monkey patching.

Fix SeqFeature not hashable error

tcmitchell · 2023-03-24T15:28:40Z

Hey @jakebeal can this one be merged now? The tests are all passing after fixing the BioPython Hashable change.

…s be 1) add a non-standard sbol#locationPosition type, and 2) reference to an unrecognized Location from a SequenceFeature. Temporarily address this by reverting to the prior behavior of truncating fuzzy ranges into ranges.

jakebeal · 2023-03-27T13:59:41Z

I am working through some errors that come up in its usage with the iGEM workflow. Once all of those are either resolved or sufficiently identified to defer into other tickets, I will merge this.

Kludge defuzz fuzzy features

…vel. The CustomReferenceProperty and CustomStructuredCommentProperty classes in the direct Genbank conversion were set up as TopLevel classes when they should have been Identified objects. This caused a great deal of problems, since their identities were still those of Identified objects. I have also changed some incorrectly plural properties to singular names, following the standard SBOL convention (e.g., the "accessions" variable is listed as a set of individual "accession" properties) After this commit, I will follow with others that clean up now-unused code and style issues.

…mTopLevel classes into behaving like CustomIdentified

In the direct SBOL3->GenBank converter, there were a number of fields only filled from saved data from GenBank->SBOL conversions Change these to default to taking information from SBOL as well: - 'molecule_type' annotation defaults based on Component.types (SBO material) - 'topology' annotation defaults based on Component.types (SO topology) - 'sequence_version' defaults to 1 - 'label' qualifier for features defaults to name Likewise, do not attempt to use missing GenBank information if it is not present. This affects annotations 'date', 'data_file_division', 'molecule_type', 'organism', 'source', 'topology', 'accessions', and 'sequence_version', and feature qualifier 'label' Other improvements: - tolerate deprecated values for orientations. - search multiple roles in component to find SO role for conversion - workaround patch for tyto URI normalization Also update tests: - Confirm that version>1 round-trips correctly - Add an sbol#inline value for an orientation - Provide diffs when GenBank conversions fail - More comprehensive test of ignoring non-converting SBOL properties - Version should always be set - No auto-generated names in converted feature (displayID is sufficient, and generating names just confuses round-tripping) Finally, style was cleaned up in various places

jakebeal · 2023-04-09T12:05:17Z

Once PR #203 and PR #206 are merged, this branch will be ready to merge.

There is still one major outstanding issue that will need to be addressed, which is the "flattening" of hierarchical Component objects to include the features of their SubComponents. I have filed that as #207, and it can be addressed separately from this PR

Fill in missing GenBank information from SBOL fields

Identified genbank customs

mohitdmak and others added 30 commits June 17, 2022 21:29

add: initial simple conversion gb/sbol3 with sample files

d3037d9

add: unittest for simple gb to sbol3 conversion

dee6e29

fix: failing unit tests when run together

37ff8b1

Update sbol_utilities/sbol3_genbank_conversion.py

86fdd71

Co-authored-by: Tom Mitchell <tcmitchell@users.noreply.github.com>

fix: code style, namespaces fix

ef030a7

Merge pull request #131 from mohitdmak/gb2sbol3__1

a5cd2a6

add: initial simple conversion gb/sbol3 with sample files

feat: move to seqIO parsing from GenBank.RecordParser

eebea02

add: parsing capabilities for genbanks with multiple records and feat…

86e1998

…ures

fix: debugs

e6061e9

add: script, csvs for GenBank, SO ontologies mappings

df93c87

fix: test namespace, script to parse ontologies file

b9c2e0d

feat: parse mappings csv, class conversion of GB

1026122

add: logging, few fixes for review

41a3605

feat: CSVs as package data into sbol_utilities

ac22361

add: orientation determination, cut/range creation for featurelocation

46ade48

change: mappings source from code to spreadsheet fix

8eb4810

Update setup.py

66dac70

Co-authored-by: Tom Mitchell <tcmitchell@users.noreply.github.com>

fix: storage of features/records in memory, code style

48104e4

Merge branch 'gb2sbol3__1' of github.com:mohitdmak/SBOL-utilities int…

ccd9768

…o gb2sbol3__1

add: ontology parsing error handling; cross platform file paths fix; …

75b4a9f

…logging; del: tmp scripts; formatting

fix: feat roles ns

971ad00

Merge pull request #135 from mohitdmak/gb2sbol3__1

a286360

add: genbank conversion support for multiple records and features

add: new converter to existing console script with fallback option to…

c472bcc

… legacy converter

move: converter selection logic inside convert_from_genbank

c7cf770

fix: raise valuerror instead of none; pr reviews

b6a965c

add: unit test for new forced conversion

9367d90

stylefix

e418e22

Merge branch 'feature/sbol3-genbank-conversion' of https://github.com…

41133b8

…/SynBioDex/SBOL-utilities into sbol3togenbank

add: basic sbol3 to genbank converter

5fb7598

add: feature parsing support for sbol3 to genbank converter

2b9d1e6

jakebeal requested changes Oct 22, 2022

View reviewed changes

fix: 0-indexed parsing of start locations by BioPython bug

34af5c4

mohitdmak mentioned this pull request Oct 23, 2022

fix: 0-indexed parsing of start locations by BioPython bug #188

Merged

mohitdmak requested a review from jakebeal October 23, 2022 04:41

jakebeal and others added 2 commits March 7, 2023 15:26

Merge pull request #188 from mohitdmak/fuzzylocs

9385171

fix: 0-indexed parsing of start locations by BioPython bug

Merge branch 'develop' into feature/sbol3-genbank-conversion

e3b9169

tcmitchell added 3 commits March 8, 2023 12:22

Fix SeqFeature not hashable error

5f6add6

SeqFeature is no longer hashable as of BioPython 1.80. Try an alternate implementation that avoids using SeqFeature instances as keys in a dict. Store the loc_positions on a new property of SeqFeature. Akin to monkey patching.

Remove unused variable

5968c11

Merge pull request #199 from SynBioDex/198-hashable-seq-feature

9637402

Fix SeqFeature not hashable error

jakebeal and others added 8 commits March 27, 2023 11:14

Merge pull request #201 from SynBioDex/200-kludge-defuzz-fuzzy-features

1840948

Kludge defuzz fuzzy features

Working on style cleanup

1d8c800

Remove unused dictionaries that had been used to kludge Genbank Custo…

93a13c3

…mTopLevel classes into behaving like CustomIdentified

More style improvements in Genbank converter file

783ce34

Style cleanup, including name refactoring, on genbank converter

08f94f3

Make Genbank converter less noisy, and do not use f-strings for logging

6f53d06

jakebeal and others added 3 commits April 11, 2023 09:11

Add documentation for tyto_normalize_term issue workaround

2989a4a

Merge pull request #206 from SynBioDex/missing-genbank-information

8680ed5

Fill in missing GenBank information from SBOL fields

Merge pull request #203 from SynBioDex/identified-genbank-customs

9337a8a

Identified genbank customs

jakebeal approved these changes Apr 11, 2023

View reviewed changes

jakebeal merged commit 2634729 into develop Apr 11, 2023

jakebeal deleted the feature/sbol3-genbank-conversion branch April 11, 2023 14:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

final merge: Offline SBOL3 <-> GenBank Converter; Integration with previous online Converter #186

final merge: Offline SBOL3 <-> GenBank Converter; Integration with previous online Converter #186

mohitdmak commented Oct 19, 2022 •

edited

Loading

jakebeal commented Oct 22, 2022

jakebeal left a comment

mohitdmak commented Oct 23, 2022 •

edited

Loading

jakebeal commented Mar 7, 2023

tcmitchell commented Mar 24, 2023

jakebeal commented Mar 27, 2023

jakebeal commented Apr 9, 2023

final merge: Offline SBOL3 <-> GenBank Converter; Integration with previous online Converter #186

final merge: Offline SBOL3 <-> GenBank Converter; Integration with previous online Converter #186

Conversation

mohitdmak commented Oct 19, 2022 • edited Loading

About the Project:

Project Blog:

Credits:

jakebeal commented Oct 22, 2022

jakebeal left a comment

Choose a reason for hiding this comment

mohitdmak commented Oct 23, 2022 • edited Loading

jakebeal commented Mar 7, 2023

tcmitchell commented Mar 24, 2023

jakebeal commented Mar 27, 2023

jakebeal commented Apr 9, 2023

mohitdmak commented Oct 19, 2022 •

edited

Loading

mohitdmak commented Oct 23, 2022 •

edited

Loading