Skip to content

Commit

Permalink
Add data export to various formats (#151)
Browse files Browse the repository at this point in the history
* add export function

* include more formats

* add docs
  • Loading branch information
lmmentel committed May 23, 2024
1 parent c0762ce commit 3945030
Show file tree
Hide file tree
Showing 4 changed files with 82 additions and 15 deletions.
24 changes: 23 additions & 1 deletion docs/source/data_access.rst
Original file line number Diff line number Diff line change
Expand Up @@ -65,4 +65,26 @@ for getting the session or the engine:

.. autofunction:: mendeleev.db.get_session

.. autofunction:: mendeleev.db.get_engine
.. autofunction:: mendeleev.db.get_engine


Export data
-----------

The data can be exported to a number of formats using the CLI by invoking `inv export` command. The following formats are supported:

- csv
- json
- html
- markdown

The command will export all the tables from the database to a set of files in the specified format.

In order to use this functionality you'll need to clone the mendeleev repository and install the package in the development mode. Here's how you can do it:

```bash
gh clone lmmentel/mendeleev
cd mendeleev
poetry install
poetry run inv export
```
27 changes: 13 additions & 14 deletions poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ sphinx-copybutton = "^0.3.1"
sphinx-material = "^0.0.32"
sphinxcontrib-bibtex = "^2.1.4"
pre-commit = "==3.5.0"
invoke = "^2.2.0"

[tool.poetry.group.vis]
optional = true
Expand Down
45 changes: 45 additions & 0 deletions tasks.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
from pathlib import Path
from invoke import task


@task
def export(c):
"""Export data to a few formats files."""
tables = [
"elements",
"groups",
"ionicradii",
"ionizationenergies",
"isotopedecaymodes",
"isotopes",
"oxidationstates",
"phasetransitions",
"screeningconstants",
"series",
]

formats = [
"csv",
"html",
"json",
"markdown",
]

for fmt in formats:
Path(f"data/{fmt}").mkdir(exist_ok=True)
for table in tables:
print(f"Exporting {table} to {fmt} ... ", end="")
c.run(
f"sqlite3 -{fmt} mendeleev/elements.db 'SELECT * FROM {table};' > data/{fmt}/{table}.{fmt}",
echo=True,
)
print("done")

# SQL dump
print("Creating SQL files with the data ... ", end="")
c.run(
"sqlite3 mendeleev/elements.db .dump > data/sql/elements.sql",
echo=True,
pty=True,
)
print("done")

0 comments on commit 3945030

Please sign in to comment.