Skip to content

Releases: GopalGB/pqtools

pqtools 0.8.0 - the container loop closes

Choose a tag to compare

@GopalGB GopalGB released this 04 Sep 06:18

Point at a .xlsx or .pbix, see the queries inside it, add one, save it back
into the file, and run it. That last half is what 0.8.0 adds.

pip install pqtools

pq list book.xlsx                    # Clients, Colors, Countries, Invoices, Stock
pq add  book.xlsx --name "Top Colours" \
    --source 'let S = #table({"Colour","N"},{{"Blue",5},{"Red",3}}) in S' --write
pq eval book.xlsx --member "Top Colours" --format csv
# Colour,N
# Blue,5
# Red,3

New

  • pq list FILE names every query in a container.
  • pq add FILE --name N --source '...' adds a new one. --write saves it in.
  • --write now works on format / rename / replace-source / add for
    containers. Every container write copies the original to <file>.bak first.
  • Local-file sources run locally: File.Contents, Csv.Document,
    Text.FromBinary, and the Binary.* family. The engine boundary is about
    capability, not the word "connector" - Sql.Database and Web.Contents
    still refuse, by design.
  • #table and #binary are registered, so a literal table written the way
    a person actually writes it now runs. #shared / #sections say they need
    the enclosing section document instead of reading like a typo.

Fixed - two bugs that had green test suites

  • UTF-16 DataMashup. Real Excel writes customXml/item1.xml as UTF-16 LE
    with a BOM. Every real workbook therefore reported "no DataMashup part
    found". The test fixture wrote UTF-8, which is why nothing caught it.
  • Quoted identifiers were never unquoted. [#"First Name"] looked up a
    field literally named #"First Name". Quiet, because step names round-tripped
    by matching themselves. Fixed at the root; containers.split_shared had its
    own copy of the same bug.

How the write path was checked

680 tests, 89% coverage, ruff + mypy + the pinned Microsoft parser bridge clean.
Container writes are verified four ways: CRC intact, every zip member except the
DataMashup part byte-identical, the M reading back, and openpyxl - an
independent xlsx parser - opening the rewritten workbook to the same sheets and
cells as the original
. Two real Excel-authored workbooks, 21,215 cells
compared, zero differences.

Residual risk, stated plainly: Excel itself has not opened a rewritten
workbook, because Excel was not installed where this was validated. That is why
every --write leaves a .bak.

Unofficial. Not affiliated with or endorsed by Microsoft. pq eval runs a
query's transformation chain locally; it is not a Power Query runtime and never
pretends to be one.

https://pypi.org/project/pqtools/0.8.0/

pqtools 0.6.0 - a real query runs with no --bind

Choose a tag to compare

@GopalGB GopalGB released this 04 Sep 04:02

Local-file sources run natively now. Csv.Document, File.Contents and
Text.FromBinary are implemented in pqtools itself, in pure Python, with no
new dependencies - so a query copied verbatim out of Power Query's Advanced
Editor, Source step included, evaluates without anything supplied.

$ pq eval report.pq --format csv
Region,Total,Orders
West,380,2
North,150,1
East,100,1

Why this changed
The old boundary treated "connector" as one indivisible thing only
Microsoft's Mashup Engine could cross. That was too broad. Sql.Database
genuinely needs an engine. Reading a local CSV is reading a local CSV, and
Csv.Document(File.Contents(...)) is the most common Source step in real
Power Query - refusing it forced everyone through --bind for the one case
that needs no help.

Added

  • Csv.Document with the options-record and positional forms, multi-character
    and whitespace delimiters, QuoteStyle.None, CsvStyle.QuoteAlways,
    ExtraValues, named/typed/counted columns, and code-page decoding with a
    UTF-8 BOM strip. It does NOT promote headers - it yields Column1..ColumnN
    and Table.PromoteHeaders stays a separate step, exactly as Power Query
    does. Auto-promoting would break every real query.
  • File.Contents, Text.FromBinary.
  • Table.ReverseRows, Table.Repeat, Table.ExpandListColumn,
    Table.SelectRowsWithErrors, Table.ReplaceErrorValues - found by probing the
    115 functions the Power Query UI actually writes. 288 builtins total.
  • evaluate, EvalError and UnsupportedError are exported from the package
    root. "from pqtools import evaluate" is the import everyone guesses first
    and it did not work; a test now enforces that every import the README
    shows resolves.

Still refused, and now named individually rather than by the word
"connector": Sql.Database, Web.Contents, SharePoint., Odbc.,
Oracle/PostgreSQL/MySQL.Database, Folder.*, Excel.Workbook. Credentials, a
network identity, driver type mapping and query folding have no honest local
stand-in, and each raises a typed error naming itself.

Deliberately still unimplemented, with reasons rather than silence:
Table.FuzzyNestedJoin (Microsoft's matching algorithm is unspecified),
Table.View (query folding with no backing source), Table.Schema (no declared
column types exist in this data model), Table.Profile and
Table.AggregateTableColumn (output column names unverified).

646 tests, 89% coverage, mypy and ruff clean.