diff --git a/README.md b/README.md index a570acb..9d1f8a4 100644 --- a/README.md +++ b/README.md @@ -9,37 +9,48 @@ The ultimate toolkit for Microsoft Excel modelers and mod-operations. #### The name -*XLtoy* it's a word pun that starts from *exel to py* concept, but the *p* seem superfluous here and *xlto(p)y* became +*XLtoy* it's a word pun that starts from **exel to py** concept, but the *p* seem superfluous here and *xlto(p)y* became XLtoy, more funny. ### Description -XLtoy framework can read, parse, diff, validate, manage changes and run out of the box complicated models written -using Microsoft Excel. Not all features are ready now, but the development plan is show below. -This tool is suitable for modelers who need change management tools, for example in a collaborative environment, -is useful know what's changed in data of formulas. No less, dev-ops (od mod-ops) than need an instrument to identify -uniquely model, data, and changes on each delivery. ---- After many year in this field, i found that is too difficult, and often useless, to analyze an entire workbook, -this approach force to write unpredictable algorithms and doesn't work because often we are interested only in a subset -of all cells. So main idea, is to identify a subset of areas of interest, defines as *working areas* -and focus only on these, so with minimum changes to an existent sheet, the parser can handle it and produce -useful information. If you can apply some simple -[rules](https://raw.githubusercontent.com/glaucouri/xltoy/main/rules.md) +this approach force to write unpredictable an inefficient algorithms and doesn't work because often we are interested only in a subset +of all cells. So main idea, is to identify a subset of areas of interest, defined as *working areas* +and focus XLtoy only on these, so with minimum changes to an existent sheet, the parser can handle it and produce +useful information. If you can apply some **simple +[rules](https://raw.githubusercontent.com/glaucouri/xltoy/main/rules.md)** you are ready to go! -This is an example of a common forecasting model that can be well handled by XLtoy. -![xlsample](https://github.com/glaucouri/xltoy/raw/main/img/simple_model.png?raw=true) -Green cells contain actual (or hystorical) values, model in salmon for the first calculated step, -and in yellow dragged cells, the rest of the model. +**So ?** + +XLtoy framework can read, parse, diff, validate, manage changes and run out of the box complicated models written +using Microsoft Excel. Not all features are ready up to now, but the development plan is show below. + +This tool is useful for users that write, share, maintain and deploy models written in Excel. Many kind of model are +well handled by XLtoy: +- validation models +- rule based models +- financial models +- forecasting models + +In a collaborative environment, for example, a change management tools, can save a lot of time and money, comparing two version +is useful know what's changed in the data or formulas. No less, dev-ops (od mod-ops) than need an instrument to identify +uniquely model, data, and changes on each delivery. Model differ can identify precisely which and where are the differences +using syntactic or semantic algorithm. ### Installation It's strongly suggested to use virtualenv: - ``` >pip3 install virtualenv >python3 -m venv XLtoy_pyenv >source XLtoy_pyenv/bin/activate +``` + +``` +>pip install xltoy + +# Or from source: >git clone https://github.com/glaucouri/XLtoy.git >cd XLtoy/ @@ -92,17 +103,20 @@ with the release of first version i will need feedback, use cases and tester. * it define [working rules](https://raw.githubusercontent.com/glaucouri/xltoy/main/rules.md) * fully testes with py3.6 to py3.8 * collector can read data,formulas and can show an entire workbook as yaml or json. -* **diff** works with data and formulas too, it can compare 2 workbook or a representation of it yaml or json. +* **diff** works with data and formulas too, it can compare 2 workbook or a representation of it yaml +or json. +* with fingerprint option model can be marked (like a md5 for a file) #### Version 0.2: parser feature: * parser can understand excel formula (probably not all syntax) * in memory graph representation with all relation between equations. * can find all predecessors and successors of a given equation. * models can be exported as graph or python code. +* execution of python version can be done in a notebook or a stand alone env. #### Version 0.3: executor feature: * data can be stored as pandas DataFrame -* models can be executed on external data. Binding feature. +* models can be binded to external data. Binding feature. and can be run on huge data set. -#### Version X: big data feature: +#### Version 0.4: big data feature: * model can be distributed on a spark cluster and executed in order to work on big data diff --git a/data/anon_sheet.parsed.yaml b/data/anon_sheet.parsed.yaml new file mode 100644 index 0000000..69973c5 --- /dev/null +++ b/data/anon_sheet.parsed.yaml @@ -0,0 +1,13 @@ +Sheet1: + anon_1: =(anon_1[T-1]+anon_1[T-2])/2 + anon_10: =LOG(anon_9[T]) + anon_11: =LOG(anon_9[T]) + anon_2: =(anon_2[T-1]+anon_2[T-2])/2 + anon_3: =(anon_3[T-1]+anon_3[T-2])/2 + anon_4: =(anon_4[T-1]+anon_4[T-2])/2 + anon_5: =if(anon_5[T-2],anon_5[T-2]+anon_5[T-1],anon_5[T-2]-anon_5[T-1]) + anon_6: =RAND() + anon_7: =anon_7[T-1]*0.023 + anon_8: =8 + anon_9: =12 + diff --git a/data/anon_sheet.xlsx b/data/anon_sheet.xlsx old mode 100755 new mode 100644 diff --git a/data/anon_sheet.yaml b/data/anon_sheet.yaml new file mode 100644 index 0000000..11f3338 --- /dev/null +++ b/data/anon_sheet.yaml @@ -0,0 +1,13 @@ +Sheet1: + anon_1: =(E3+D3)/2 + anon_10: =LOG(F11) + anon_11: =LOG($F$11) + anon_2: =(E4+D4)/2 + anon_3: =(E5+D5)/2 + anon_4: =(E6+D6)/2 + anon_5: =IF(D7,D7+E7,D7-E7) + anon_6: =RAND() + anon_7: =E9*0.023 + anon_8: =8 + anon_9: =12 + diff --git a/data/data_only.parsed.yaml b/data/data_only.parsed.yaml new file mode 100644 index 0000000..7d11644 --- /dev/null +++ b/data/data_only.parsed.yaml @@ -0,0 +1,290 @@ +data_she: + B10: 0.5541456386323022 + B11: 0.026085289680903734 + B12: 0.9767405888667278 + B13: 0.21291005166704624 + B14: 0.037980083896950845 + B15: 0.5960076338820124 + B16: 0.6464208635679233 + B17: 0.37476823089793565 + B18: 0.12332809081099005 + B19: 0.6929184236394574 + B20: 0.4893838195112623 + B3: 0.9153213929911252 + B4: 0.24500196840142907 + B5: 0.4861065256701226 + B6: 0.6230952346247797 + B7: 0.4265166025889605 + B8: 0.7571404094622627 + B9: 0.9584693165926973 + C10: 0.11352298560091323 + C11: 0.8164185598990812 + C12: 0.8362466653063508 + C13: 0.19792854804318238 + C14: 0.6838543023264179 + C15: 0.5579580568628854 + C16: 0.9828331547666131 + C17: 0.7913733765331421 + C18: 0.8534036785557345 + C19: 0.09291836967544309 + C20: 0.5221955537153197 + C3: 0.02997109776645901 + C4: 0.1695982635664086 + C5: 0.9027771679725788 + C6: 0.9357789541551214 + C7: 0.5750368733891794 + C8: 0.18838131728488772 + C9: 0.7983314389966031 + D10: 0.36215022143100295 + D11: 0.6272466550806207 + D12: 0.010357255542100874 + D13: 0.28117023174852895 + D14: 0.16257902941353897 + D15: 0.2276186822224523 + D16: 0.20215713991324458 + D17: 0.3463483224572296 + D18: 0.9469536106565644 + D19: 0.39781113147132596 + D20: 0.8987470214411022 + D3: 0.2942586336828905 + D4: 0.9876158286360819 + D5: 0.026056063943006302 + D6: 0.4522437242934847 + D7: 0.9939792507410259 + D8: 0.1322997405993923 + D9: 0.4171024463533982 + E10: 0.6127984070035475 + E11: 0.2054105708021966 + E12: 0.6426284773284412 + E13: 0.8841689589413451 + E14: 0.6869673837540939 + E15: 0.36311655967191936 + E16: 0.8204500664474473 + E17: 0.7098324526907621 + E18: 0.1940069146812139 + E19: 0.7812864209658983 + E20: 0.6241738565327017 + E3: 0.997112758253685 + E4: 0.2606652242435724 + E5: 0.447788668253979 + E6: 0.9571647882935546 + E7: 0.6727398406081185 + E8: 0.3741351782523885 + E9: 0.6951559656090865 + F10: 0.5034113433677058 + F11: 0.22045144558761953 + F12: 0.10334601968877644 + F13: 0.8906660474590936 + F14: 0.8201781137204901 + F15: 0.46745103744374883 + F16: 0.7757241287934525 + F17: 0.4624450334096937 + F18: 0.6746516305391441 + F19: 0.18243740586962487 + F20: 0.587740077215051 + F3: 0.038922504566889726 + F4: 0.8800394646955587 + F5: 0.925549939246955 + F6: 0.623822257905231 + F7: 0.5126002780706348 + F8: 0.6007754021271984 + F9: 0.3568683528483584 + G10: 0.2860283467812502 + G11: 0.4809368101252425 + G12: 0.5412339659573502 + G13: 0.5829086794567515 + G14: 0.7927666908740054 + G15: 0.7283112397956267 + G16: 0.6546451667462461 + G17: 0.30833946194849815 + G18: 0.6568215981049226 + G19: 0.10146737613297796 + G20: 0.26601568408550846 + G3: 0.3552541616445003 + G4: 0.8812917085462547 + G5: 0.45584927288855714 + G6: 0.6997538957282622 + G7: 0.9631783926809406 + G8: 0.8282552925349292 + G9: 0.052873692699810126 + H10: 0.3383220009061745 + H11: 0.6433172670085517 + H12: 0.891393662748517 + H13: 0.5273656824598986 + H14: 0.5775062566202573 + H15: 0.7469021446132771 + H16: 0.4917375721226541 + H17: 0.35024080990911843 + H18: 0.7373328397698357 + H19: 0.7599030937800939 + H20: 0.5955243531186635 + H3: 0.2928400931333909 + H4: 0.46949601739232927 + H5: 0.272197607038839 + H6: 0.9473819349782844 + H7: 0.7262160991382033 + H8: 0.5621639665651894 + H9: 0.6859652306991587 + I10: 0.4394086544839796 + I11: 0.40983125540417853 + I12: 0.4013865876591758 + I13: 0.08916563495191898 + I14: 0.09682205787418419 + I15: 0.7083158305243311 + I16: 0.435953083132108 + I17: 0.07162741026594577 + I18: 0.9049304609729177 + I19: 0.7709980976746283 + I20: 0.0005099366435238339 + I3: 0.8916407491527066 + I4: 0.9589079862146723 + I5: 0.8368556781298907 + I6: 0.10637163586021903 + I7: 0.9364720832553128 + I8: 0.4296498112400442 + I9: 0.7497805392311474 + J10: 0.12362420434897181 + J11: 0.9425077725303377 + J12: 0.36382655255333307 + J13: 0.9047113587608068 + J14: 0.6793802859351581 + J15: 0.86553818776169 + J16: 0.0069396530207105345 + J17: 0.8782045807042917 + J18: 0.6431706230305294 + J19: 0.3479485516785956 + J20: 0.7616353411421168 + J3: 0.26701397927673587 + J4: 0.26298667096495887 + J5: 0.19760646370231794 + J6: 0.23911218997525352 + J7: 0.3453318388416696 + J8: 0.6385747833336318 + J9: 0.8290041053454821 + K10: 0.6694850566908758 + K11: 0.5119256403940853 + K12: 0.5397996840942567 + K13: 0.5587617149870788 + K14: 0.25553731558477155 + K15: 0.25436592512601275 + K16: 0.6076992994472725 + K17: 0.5478498031976389 + K18: 0.6342495630237412 + K19: 0.8518045833597062 + K20: 0.28671671425402934 + K3: 0.908094556132203 + K4: 0.6924531714377297 + K5: 0.7043006724099063 + K6: 0.9676652666560044 + K7: 0.4710143755869326 + K8: 0.8382192634013961 + K9: 0.7112030999881722 + L10: 0.37813718135572383 + L11: 0.733804434123833 + L12: 0.5969589645453606 + L13: 0.8917995333078093 + L14: 0.7192474885259087 + L15: 0.6449241285407299 + L16: 0.34583244808442326 + L17: 0.7126947043850906 + L18: 0.75657419437274 + L19: 0.34974114445672855 + L20: 0.27889534186976905 + L3: 0.3046018233007758 + L4: 0.38400038718653884 + L5: 0.9153388476603972 + L6: 0.6646131109723192 + L7: 0.2159737578371469 + L8: 0.13780175681145557 + L9: 0.9836970148602205 + M10: 0.40297295074509265 + M11: 0.0011042053194513235 + M12: 0.139416309741774 + M13: 0.7604937322735423 + M14: 0.3077570204232998 + M15: 0.6837499784415718 + M16: 0.30676247430602066 + M17: 0.21101006215731222 + M18: 0.554367864339741 + M19: 0.1888834316822553 + M20: 0.8574004012222576 + M3: 0.9982418889514836 + M4: 0.32913364806140744 + M5: 0.9124968272660314 + M6: 0.3251774904764252 + M7: 0.7977391800995233 + M8: 0.8029886192525919 + M9: 0.643618429660081 + N10: 0.7075685246001349 + N11: 0.8037094223954033 + N12: 0.42224109016597 + N13: 0.4632250921878892 + N14: 0.856433485299263 + N15: 0.10169864061929279 + N16: 0.883717168296255 + N17: 0.7997105012268015 + N18: 0.9384600507761541 + N19: 0.6256552171071543 + N20: 0.3121218474644948 + N3: 0.29995886228307556 + N4: 0.6158394243663705 + N5: 0.059240850587369165 + N6: 0.5433765331336373 + N7: 0.06964341328610857 + N8: 0.8289887978812367 + N9: 0.5133603151311821 + O10: 0.3530894027007806 + O11: 0.8031396254172442 + O12: 0.2812473521150449 + O13: 0.382763788911775 + O14: 0.8180716772947578 + O15: 0.28212520138451447 + O16: 0.717926893213031 + O17: 0.36973427503995326 + O18: 0.01359971616806066 + O19: 0.30869192970576464 + O20: 0.7069108913968395 + O3: 0.9942776833708358 + O4: 0.075201097374898 + O5: 0.8035243755813781 + O6: 0.3752802256652822 + O7: 0.283165834348593 + O8: 0.7698255384051332 + O9: 0.6831978293346382 + P10: 0.2545178015578923 + P11: 0.6198354108691334 + P12: 0.22955728923835172 + P13: 0.382955103324258 + P14: 0.23631242260465024 + P15: 0.995429885292081 + P16: 0.679451279301183 + P17: 0.7816726348232035 + P18: 0.43596965291003853 + P19: 0.8802863097098516 + P20: 0.5836179943407604 + P3: 0.11955118835825396 + P4: 0.5546936069090002 + P5: 0.05163951386370236 + P6: 0.44145328098309866 + P7: 0.22517261847602155 + P8: 0.057439778103524075 + P9: 0.15362086188001012 + Q10: 0.06896170452638406 + Q11: 0.06743346259422855 + Q12: 0.17721790431127882 + Q13: 0.6061917720178265 + Q14: 0.3285847620350022 + Q15: 0.2404021650155972 + Q16: 0.9876157409378274 + Q17: 0.35679922880570525 + Q18: 0.22020127097700948 + Q19: 0.9667535457636863 + Q20: 0.6353564475076718 + Q3: 0.7513083354504116 + Q4: 0.7970686798766001 + Q5: 0.3914063620847299 + Q6: 0.8410707253371789 + Q7: 0.603777122295439 + Q8: 0.9279215983751176 + Q9: 0.2629755374584216 + diff --git a/data/data_only.xlsx b/data/data_only.xlsx old mode 100755 new mode 100644 diff --git a/data/data_only.yaml b/data/data_only.yaml new file mode 100644 index 0000000..7d11644 --- /dev/null +++ b/data/data_only.yaml @@ -0,0 +1,290 @@ +data_she: + B10: 0.5541456386323022 + B11: 0.026085289680903734 + B12: 0.9767405888667278 + B13: 0.21291005166704624 + B14: 0.037980083896950845 + B15: 0.5960076338820124 + B16: 0.6464208635679233 + B17: 0.37476823089793565 + B18: 0.12332809081099005 + B19: 0.6929184236394574 + B20: 0.4893838195112623 + B3: 0.9153213929911252 + B4: 0.24500196840142907 + B5: 0.4861065256701226 + B6: 0.6230952346247797 + B7: 0.4265166025889605 + B8: 0.7571404094622627 + B9: 0.9584693165926973 + C10: 0.11352298560091323 + C11: 0.8164185598990812 + C12: 0.8362466653063508 + C13: 0.19792854804318238 + C14: 0.6838543023264179 + C15: 0.5579580568628854 + C16: 0.9828331547666131 + C17: 0.7913733765331421 + C18: 0.8534036785557345 + C19: 0.09291836967544309 + C20: 0.5221955537153197 + C3: 0.02997109776645901 + C4: 0.1695982635664086 + C5: 0.9027771679725788 + C6: 0.9357789541551214 + C7: 0.5750368733891794 + C8: 0.18838131728488772 + C9: 0.7983314389966031 + D10: 0.36215022143100295 + D11: 0.6272466550806207 + D12: 0.010357255542100874 + D13: 0.28117023174852895 + D14: 0.16257902941353897 + D15: 0.2276186822224523 + D16: 0.20215713991324458 + D17: 0.3463483224572296 + D18: 0.9469536106565644 + D19: 0.39781113147132596 + D20: 0.8987470214411022 + D3: 0.2942586336828905 + D4: 0.9876158286360819 + D5: 0.026056063943006302 + D6: 0.4522437242934847 + D7: 0.9939792507410259 + D8: 0.1322997405993923 + D9: 0.4171024463533982 + E10: 0.6127984070035475 + E11: 0.2054105708021966 + E12: 0.6426284773284412 + E13: 0.8841689589413451 + E14: 0.6869673837540939 + E15: 0.36311655967191936 + E16: 0.8204500664474473 + E17: 0.7098324526907621 + E18: 0.1940069146812139 + E19: 0.7812864209658983 + E20: 0.6241738565327017 + E3: 0.997112758253685 + E4: 0.2606652242435724 + E5: 0.447788668253979 + E6: 0.9571647882935546 + E7: 0.6727398406081185 + E8: 0.3741351782523885 + E9: 0.6951559656090865 + F10: 0.5034113433677058 + F11: 0.22045144558761953 + F12: 0.10334601968877644 + F13: 0.8906660474590936 + F14: 0.8201781137204901 + F15: 0.46745103744374883 + F16: 0.7757241287934525 + F17: 0.4624450334096937 + F18: 0.6746516305391441 + F19: 0.18243740586962487 + F20: 0.587740077215051 + F3: 0.038922504566889726 + F4: 0.8800394646955587 + F5: 0.925549939246955 + F6: 0.623822257905231 + F7: 0.5126002780706348 + F8: 0.6007754021271984 + F9: 0.3568683528483584 + G10: 0.2860283467812502 + G11: 0.4809368101252425 + G12: 0.5412339659573502 + G13: 0.5829086794567515 + G14: 0.7927666908740054 + G15: 0.7283112397956267 + G16: 0.6546451667462461 + G17: 0.30833946194849815 + G18: 0.6568215981049226 + G19: 0.10146737613297796 + G20: 0.26601568408550846 + G3: 0.3552541616445003 + G4: 0.8812917085462547 + G5: 0.45584927288855714 + G6: 0.6997538957282622 + G7: 0.9631783926809406 + G8: 0.8282552925349292 + G9: 0.052873692699810126 + H10: 0.3383220009061745 + H11: 0.6433172670085517 + H12: 0.891393662748517 + H13: 0.5273656824598986 + H14: 0.5775062566202573 + H15: 0.7469021446132771 + H16: 0.4917375721226541 + H17: 0.35024080990911843 + H18: 0.7373328397698357 + H19: 0.7599030937800939 + H20: 0.5955243531186635 + H3: 0.2928400931333909 + H4: 0.46949601739232927 + H5: 0.272197607038839 + H6: 0.9473819349782844 + H7: 0.7262160991382033 + H8: 0.5621639665651894 + H9: 0.6859652306991587 + I10: 0.4394086544839796 + I11: 0.40983125540417853 + I12: 0.4013865876591758 + I13: 0.08916563495191898 + I14: 0.09682205787418419 + I15: 0.7083158305243311 + I16: 0.435953083132108 + I17: 0.07162741026594577 + I18: 0.9049304609729177 + I19: 0.7709980976746283 + I20: 0.0005099366435238339 + I3: 0.8916407491527066 + I4: 0.9589079862146723 + I5: 0.8368556781298907 + I6: 0.10637163586021903 + I7: 0.9364720832553128 + I8: 0.4296498112400442 + I9: 0.7497805392311474 + J10: 0.12362420434897181 + J11: 0.9425077725303377 + J12: 0.36382655255333307 + J13: 0.9047113587608068 + J14: 0.6793802859351581 + J15: 0.86553818776169 + J16: 0.0069396530207105345 + J17: 0.8782045807042917 + J18: 0.6431706230305294 + J19: 0.3479485516785956 + J20: 0.7616353411421168 + J3: 0.26701397927673587 + J4: 0.26298667096495887 + J5: 0.19760646370231794 + J6: 0.23911218997525352 + J7: 0.3453318388416696 + J8: 0.6385747833336318 + J9: 0.8290041053454821 + K10: 0.6694850566908758 + K11: 0.5119256403940853 + K12: 0.5397996840942567 + K13: 0.5587617149870788 + K14: 0.25553731558477155 + K15: 0.25436592512601275 + K16: 0.6076992994472725 + K17: 0.5478498031976389 + K18: 0.6342495630237412 + K19: 0.8518045833597062 + K20: 0.28671671425402934 + K3: 0.908094556132203 + K4: 0.6924531714377297 + K5: 0.7043006724099063 + K6: 0.9676652666560044 + K7: 0.4710143755869326 + K8: 0.8382192634013961 + K9: 0.7112030999881722 + L10: 0.37813718135572383 + L11: 0.733804434123833 + L12: 0.5969589645453606 + L13: 0.8917995333078093 + L14: 0.7192474885259087 + L15: 0.6449241285407299 + L16: 0.34583244808442326 + L17: 0.7126947043850906 + L18: 0.75657419437274 + L19: 0.34974114445672855 + L20: 0.27889534186976905 + L3: 0.3046018233007758 + L4: 0.38400038718653884 + L5: 0.9153388476603972 + L6: 0.6646131109723192 + L7: 0.2159737578371469 + L8: 0.13780175681145557 + L9: 0.9836970148602205 + M10: 0.40297295074509265 + M11: 0.0011042053194513235 + M12: 0.139416309741774 + M13: 0.7604937322735423 + M14: 0.3077570204232998 + M15: 0.6837499784415718 + M16: 0.30676247430602066 + M17: 0.21101006215731222 + M18: 0.554367864339741 + M19: 0.1888834316822553 + M20: 0.8574004012222576 + M3: 0.9982418889514836 + M4: 0.32913364806140744 + M5: 0.9124968272660314 + M6: 0.3251774904764252 + M7: 0.7977391800995233 + M8: 0.8029886192525919 + M9: 0.643618429660081 + N10: 0.7075685246001349 + N11: 0.8037094223954033 + N12: 0.42224109016597 + N13: 0.4632250921878892 + N14: 0.856433485299263 + N15: 0.10169864061929279 + N16: 0.883717168296255 + N17: 0.7997105012268015 + N18: 0.9384600507761541 + N19: 0.6256552171071543 + N20: 0.3121218474644948 + N3: 0.29995886228307556 + N4: 0.6158394243663705 + N5: 0.059240850587369165 + N6: 0.5433765331336373 + N7: 0.06964341328610857 + N8: 0.8289887978812367 + N9: 0.5133603151311821 + O10: 0.3530894027007806 + O11: 0.8031396254172442 + O12: 0.2812473521150449 + O13: 0.382763788911775 + O14: 0.8180716772947578 + O15: 0.28212520138451447 + O16: 0.717926893213031 + O17: 0.36973427503995326 + O18: 0.01359971616806066 + O19: 0.30869192970576464 + O20: 0.7069108913968395 + O3: 0.9942776833708358 + O4: 0.075201097374898 + O5: 0.8035243755813781 + O6: 0.3752802256652822 + O7: 0.283165834348593 + O8: 0.7698255384051332 + O9: 0.6831978293346382 + P10: 0.2545178015578923 + P11: 0.6198354108691334 + P12: 0.22955728923835172 + P13: 0.382955103324258 + P14: 0.23631242260465024 + P15: 0.995429885292081 + P16: 0.679451279301183 + P17: 0.7816726348232035 + P18: 0.43596965291003853 + P19: 0.8802863097098516 + P20: 0.5836179943407604 + P3: 0.11955118835825396 + P4: 0.5546936069090002 + P5: 0.05163951386370236 + P6: 0.44145328098309866 + P7: 0.22517261847602155 + P8: 0.057439778103524075 + P9: 0.15362086188001012 + Q10: 0.06896170452638406 + Q11: 0.06743346259422855 + Q12: 0.17721790431127882 + Q13: 0.6061917720178265 + Q14: 0.3285847620350022 + Q15: 0.2404021650155972 + Q16: 0.9876157409378274 + Q17: 0.35679922880570525 + Q18: 0.22020127097700948 + Q19: 0.9667535457636863 + Q20: 0.6353564475076718 + Q3: 0.7513083354504116 + Q4: 0.7970686798766001 + Q5: 0.3914063620847299 + Q6: 0.8410707253371789 + Q7: 0.603777122295439 + Q8: 0.9279215983751176 + Q9: 0.2629755374584216 + diff --git a/data/data_sample1.parsed.yaml b/data/data_sample1.parsed.yaml new file mode 100644 index 0000000..ff5eb21 --- /dev/null +++ b/data/data_sample1.parsed.yaml @@ -0,0 +1,130 @@ +data_sample: + D10: 507579.5 + D11: 2.540113925933838 + D4: 243.58688354492188 + D5: 981.8154296875 + D6: 974.3475341796875 + D7: 671326 + D8: null + D9: 417895.8125 + E10: 467265 + E11: 2.556509494781494 + E4: 242.791015625 + E5: 976.6210327148438 + E6: 971.1640625 + E7: 659110 + E8: null + E9: 382527.28125 + F10: 480897.5 + F11: 2.54644775390625 + F4: 245.53134155273438 + F5: 979.336181640625 + F6: 982.1253662109375 + F7: 657656 + F8: null + F9: 383177.8125 + G10: 499018.5 + G11: 2.531107902526856 + G4: 234.80374145507812 + G5: 966.7130126953125 + G6: 939.2149658203125 + G7: 698109 + G8: 83819.296875 + G9: 376420.0625 + H10: 472664.5 + H11: 2.542474031448364 + H4: 243.1981658935547 + H5: 966.32421875 + H6: 972.7926635742188 + H7: 723673 + H8: 82227.984375 + H9: 368261.6875 + I10: 469867.5 + I11: 2.551462650299072 + I4: 245.06390380859375 + I5: 968.59716796875 + I6: 980.255615234375 + I7: 742352 + I8: 80773.09375 + I9: 363038.25 + J10: 452536.5 + J11: 2.574562549591064 + J4: 239.26527404785156 + J5: 962.3310546875 + J6: 957.0610961914062 + J7: 773681 + J8: 79567.2265625 + J9: 368659 + K10: 462964 + K11: 2.564170598983765 + K4: 240.69418334960938 + K5: 968.2215576171875 + K6: 962.7767333984375 + K7: 850985 + K8: 79070.2421875 + K9: 380063.65625 + L10: 486379 + L11: 2.576596736907959 + L4: 240.44053649902344 + L5: 965.4638671875 + L6: 961.7621459960938 + L7: 888077 + L8: 79706.90625 + L9: 392281.65625 + M10: 473957.46875 + M11: 2.564022302627564 + M4: 234.94024658203125 + M5: 955.3402099609375 + M6: 939.760986328125 + M7: 905509 + M8: 79520.09375 + M9: 376665.5 + N10: 436266 + N11: 2.563867330551148 + N4: 240.65753173828125 + N5: 956.7325439453125 + N6: 962.630126953125 + N7: 910074 + N8: 78656.234375 + N9: 367932.59375 + O10: 418456 + O11: 2.596183776855469 + O4: 244.769287109375 + O5: 960.8076171875 + O6: 979.0771484375 + O7: 864497 + O8: 82552.578125 + O9: 335549.75 + P10: 423065.5 + P11: 2.636713027954102 + P4: 239.0762481689453 + P5: 959.4432983398438 + P6: 956.3049926757812 + P7: 964735 + P8: 83926.0078125 + P9: 353485.84375 + Q10: 375459 + Q11: 2.628951072692871 + Q4: 237.9830780029297 + Q5: 962.4861450195312 + Q6: 951.9323120117188 + Q7: 951298 + Q8: 85729.078125 + Q9: 333017.28125 + R10: 360714 + R11: 2.681182622909546 + R4: 239.9556121826172 + R5: 961.7842407226562 + R6: 959.8224487304688 + R7: 932617 + R8: 88160.1953125 + R9: 324605.5 + S10: 367432.5 + S11: 2.690957069396973 + S4: 238.0694122314453 + S5: 955.0843505859375 + S6: 952.2776489257812 + S7: 920347 + S8: 90300.109375 + S9: 328295.46875 + diff --git a/data/data_sample1.xlsx b/data/data_sample1.xlsx old mode 100755 new mode 100644 diff --git a/data/data_sample1.yaml b/data/data_sample1.yaml new file mode 100644 index 0000000..ff5eb21 --- /dev/null +++ b/data/data_sample1.yaml @@ -0,0 +1,130 @@ +data_sample: + D10: 507579.5 + D11: 2.540113925933838 + D4: 243.58688354492188 + D5: 981.8154296875 + D6: 974.3475341796875 + D7: 671326 + D8: null + D9: 417895.8125 + E10: 467265 + E11: 2.556509494781494 + E4: 242.791015625 + E5: 976.6210327148438 + E6: 971.1640625 + E7: 659110 + E8: null + E9: 382527.28125 + F10: 480897.5 + F11: 2.54644775390625 + F4: 245.53134155273438 + F5: 979.336181640625 + F6: 982.1253662109375 + F7: 657656 + F8: null + F9: 383177.8125 + G10: 499018.5 + G11: 2.531107902526856 + G4: 234.80374145507812 + G5: 966.7130126953125 + G6: 939.2149658203125 + G7: 698109 + G8: 83819.296875 + G9: 376420.0625 + H10: 472664.5 + H11: 2.542474031448364 + H4: 243.1981658935547 + H5: 966.32421875 + H6: 972.7926635742188 + H7: 723673 + H8: 82227.984375 + H9: 368261.6875 + I10: 469867.5 + I11: 2.551462650299072 + I4: 245.06390380859375 + I5: 968.59716796875 + I6: 980.255615234375 + I7: 742352 + I8: 80773.09375 + I9: 363038.25 + J10: 452536.5 + J11: 2.574562549591064 + J4: 239.26527404785156 + J5: 962.3310546875 + J6: 957.0610961914062 + J7: 773681 + J8: 79567.2265625 + J9: 368659 + K10: 462964 + K11: 2.564170598983765 + K4: 240.69418334960938 + K5: 968.2215576171875 + K6: 962.7767333984375 + K7: 850985 + K8: 79070.2421875 + K9: 380063.65625 + L10: 486379 + L11: 2.576596736907959 + L4: 240.44053649902344 + L5: 965.4638671875 + L6: 961.7621459960938 + L7: 888077 + L8: 79706.90625 + L9: 392281.65625 + M10: 473957.46875 + M11: 2.564022302627564 + M4: 234.94024658203125 + M5: 955.3402099609375 + M6: 939.760986328125 + M7: 905509 + M8: 79520.09375 + M9: 376665.5 + N10: 436266 + N11: 2.563867330551148 + N4: 240.65753173828125 + N5: 956.7325439453125 + N6: 962.630126953125 + N7: 910074 + N8: 78656.234375 + N9: 367932.59375 + O10: 418456 + O11: 2.596183776855469 + O4: 244.769287109375 + O5: 960.8076171875 + O6: 979.0771484375 + O7: 864497 + O8: 82552.578125 + O9: 335549.75 + P10: 423065.5 + P11: 2.636713027954102 + P4: 239.0762481689453 + P5: 959.4432983398438 + P6: 956.3049926757812 + P7: 964735 + P8: 83926.0078125 + P9: 353485.84375 + Q10: 375459 + Q11: 2.628951072692871 + Q4: 237.9830780029297 + Q5: 962.4861450195312 + Q6: 951.9323120117188 + Q7: 951298 + Q8: 85729.078125 + Q9: 333017.28125 + R10: 360714 + R11: 2.681182622909546 + R4: 239.9556121826172 + R5: 961.7842407226562 + R6: 959.8224487304688 + R7: 932617 + R8: 88160.1953125 + R9: 324605.5 + S10: 367432.5 + S11: 2.690957069396973 + S4: 238.0694122314453 + S5: 955.0843505859375 + S6: 952.2776489257812 + S7: 920347 + S8: 90300.109375 + S9: 328295.46875 + diff --git a/data/data_sample1_diff.parsed.yaml b/data/data_sample1_diff.parsed.yaml new file mode 100644 index 0000000..9f45d4b --- /dev/null +++ b/data/data_sample1_diff.parsed.yaml @@ -0,0 +1,146 @@ +data_sample: + D10: 507579.5 + D11: 2.540113925933838 + D12: 0.0005004366657703548 + D4: 243.58688354492188 + D5: 981.8154296875 + D6: 974.3475341796875 + D7: 671326 + D8: null + D9: 417895.8125 + E10: 467265 + E11: 2.556509494781494 + E12: 0.000547121974635698 + E4: 242.791015625 + E5: 976.6210327148438 + E6: 971.1640625 + E7: 659110 + E8: null + E9: 382527.28125 + F10: 480897.5 + F11: 2.54644775390625 + F12: 0.0005295198569146752 + F4: 245.53134155273438 + F5: 979.336181640625 + F6: 982.1253662109375 + F7: 657656 + F8: null + F9: 383177.8125 + G10: 499018.5 + G11: 2.531107902526856 + G12: 0.0005072172479631228 + G4: 234.80374145507812 + G5: 966.7130126953125 + G6: 939.2149658203125 + G7: 698109 + G8: 83819.296875 + G9: 376420.0625 + H10: 472664.5 + H11: 2.542474031448364 + H12: 0.0005379024723558388 + H4: 243.1981658935547 + H5: 966.32421875 + H6: 972.7926635742188 + H7: 723673 + H8: 82227.984375 + H9: 368261.6875 + I10: 469867.5 + I11: 2.551462650299072 + I12: 0.0005430174783953075 + I4: 245.06390380859375 + I5: 968.59716796875 + I6: 980.255615234375 + I7: 742352 + I8: 80773.09375 + I9: 363038.25 + J10: 452536.5 + J11: 2.574562549591064 + J12: 0.0005689182087171011 + J4: 239.26527404785156 + J5: 962.3310546875 + J6: 957.0610961914062 + J7: 773681 + J8: 79567.2265625 + J9: 368659 + K10: 462964 + K11: 2.564170598983765 + K12: 0.0005538596087349697 + K4: 240.69418334960938 + K5: 968.2215576171875 + K6: 962.7767333984375 + K7: 850985 + K8: 79070.2421875 + K9: 380063.65625 + L10: 486379 + L11: 2.576596736907959 + L12: 0.0005297508191981888 + L4: 240.44053649902344 + L5: 965.4638671875 + L6: 961.7621459960938 + L7: 888077 + L8: 79706.90625 + L9: 392281.65625 + M10: 473957.46875 + M11: 2.564022302627564 + M12: 0.0005409815166305603 + M4: 234.94024658203125 + M5: 955.3402099609375 + M6: 939.760986328125 + M7: 905510 + M8: 79520.09375 + M9: 376665.5 + N10: 436266 + N11: 2.563867330551148 + N12: 0.0005876844243079103 + N4: 240.65753173828125 + N5: 956.7325439453125 + N6: 962.630126953125 + N7: 910074 + N8: 78656.234375 + N9: 367932.59375 + O10: 418456 + O11: 2.596183776855469 + O12: 0.0006204197757602877 + O4: 244.769287109375 + O5: 960.8076171875 + O6: 979.0771484375 + O7: 864497 + O8: 82552.578125 + O9: 335549.75 + P10: 423065.5 + P11: 2.636713027954102 + P12: 0.0006232399068120899 + P4: 239.0762481689453 + P5: 959.4432983398438 + P6: 956.3049926757812 + P7: 964735 + P8: 83926.0078125 + P9: 353485.84375 + Q10: 375459 + Q11: 2.628951072692871 + Q12: 0.0007001965787723482 + Q4: 237.9830780029297 + Q5: 962.4861450195312 + Q6: 951.9323120117188 + Q7: 951298 + Q8: 85729.078125 + Q9: 333017.28125 + R10: 360714 + R11: 2.681182622909546 + R12: 0.0007432987416372932 + R4: 239.9556121826172 + R5: 961.7842407226562 + R6: 959.8224487304688 + R7: 932617 + R8: 88160.1953125 + R9: 324605.5 + S10: 367432.5 + S11: 2.690957069396973 + S12: 0.0007323677326847715 + S4: 238.0694122314453 + S5: 955.0843505859375 + S6: 952.2776489257812 + S7: 920347 + S8: 90300.109375 + S9: 328295.46875 + diff --git a/data/data_sample1_diff.xlsx b/data/data_sample1_diff.xlsx old mode 100755 new mode 100644 diff --git a/data/data_sample1_diff.yaml b/data/data_sample1_diff.yaml new file mode 100644 index 0000000..9f45d4b --- /dev/null +++ b/data/data_sample1_diff.yaml @@ -0,0 +1,146 @@ +data_sample: + D10: 507579.5 + D11: 2.540113925933838 + D12: 0.0005004366657703548 + D4: 243.58688354492188 + D5: 981.8154296875 + D6: 974.3475341796875 + D7: 671326 + D8: null + D9: 417895.8125 + E10: 467265 + E11: 2.556509494781494 + E12: 0.000547121974635698 + E4: 242.791015625 + E5: 976.6210327148438 + E6: 971.1640625 + E7: 659110 + E8: null + E9: 382527.28125 + F10: 480897.5 + F11: 2.54644775390625 + F12: 0.0005295198569146752 + F4: 245.53134155273438 + F5: 979.336181640625 + F6: 982.1253662109375 + F7: 657656 + F8: null + F9: 383177.8125 + G10: 499018.5 + G11: 2.531107902526856 + G12: 0.0005072172479631228 + G4: 234.80374145507812 + G5: 966.7130126953125 + G6: 939.2149658203125 + G7: 698109 + G8: 83819.296875 + G9: 376420.0625 + H10: 472664.5 + H11: 2.542474031448364 + H12: 0.0005379024723558388 + H4: 243.1981658935547 + H5: 966.32421875 + H6: 972.7926635742188 + H7: 723673 + H8: 82227.984375 + H9: 368261.6875 + I10: 469867.5 + I11: 2.551462650299072 + I12: 0.0005430174783953075 + I4: 245.06390380859375 + I5: 968.59716796875 + I6: 980.255615234375 + I7: 742352 + I8: 80773.09375 + I9: 363038.25 + J10: 452536.5 + J11: 2.574562549591064 + J12: 0.0005689182087171011 + J4: 239.26527404785156 + J5: 962.3310546875 + J6: 957.0610961914062 + J7: 773681 + J8: 79567.2265625 + J9: 368659 + K10: 462964 + K11: 2.564170598983765 + K12: 0.0005538596087349697 + K4: 240.69418334960938 + K5: 968.2215576171875 + K6: 962.7767333984375 + K7: 850985 + K8: 79070.2421875 + K9: 380063.65625 + L10: 486379 + L11: 2.576596736907959 + L12: 0.0005297508191981888 + L4: 240.44053649902344 + L5: 965.4638671875 + L6: 961.7621459960938 + L7: 888077 + L8: 79706.90625 + L9: 392281.65625 + M10: 473957.46875 + M11: 2.564022302627564 + M12: 0.0005409815166305603 + M4: 234.94024658203125 + M5: 955.3402099609375 + M6: 939.760986328125 + M7: 905510 + M8: 79520.09375 + M9: 376665.5 + N10: 436266 + N11: 2.563867330551148 + N12: 0.0005876844243079103 + N4: 240.65753173828125 + N5: 956.7325439453125 + N6: 962.630126953125 + N7: 910074 + N8: 78656.234375 + N9: 367932.59375 + O10: 418456 + O11: 2.596183776855469 + O12: 0.0006204197757602877 + O4: 244.769287109375 + O5: 960.8076171875 + O6: 979.0771484375 + O7: 864497 + O8: 82552.578125 + O9: 335549.75 + P10: 423065.5 + P11: 2.636713027954102 + P12: 0.0006232399068120899 + P4: 239.0762481689453 + P5: 959.4432983398438 + P6: 956.3049926757812 + P7: 964735 + P8: 83926.0078125 + P9: 353485.84375 + Q10: 375459 + Q11: 2.628951072692871 + Q12: 0.0007001965787723482 + Q4: 237.9830780029297 + Q5: 962.4861450195312 + Q6: 951.9323120117188 + Q7: 951298 + Q8: 85729.078125 + Q9: 333017.28125 + R10: 360714 + R11: 2.681182622909546 + R12: 0.0007432987416372932 + R4: 239.9556121826172 + R5: 961.7842407226562 + R6: 959.8224487304688 + R7: 932617 + R8: 88160.1953125 + R9: 324605.5 + S10: 367432.5 + S11: 2.690957069396973 + S12: 0.0007323677326847715 + S4: 238.0694122314453 + S5: 955.0843505859375 + S6: 952.2776489257812 + S7: 920347 + S8: 90300.109375 + S9: 328295.46875 + diff --git a/data/data_sample1_relative.parsed.yaml b/data/data_sample1_relative.parsed.yaml new file mode 100644 index 0000000..2d81130 --- /dev/null +++ b/data/data_sample1_relative.parsed.yaml @@ -0,0 +1,130 @@ +data_sample: + AA17: 240.44053649902344 + AA18: 965.4638671875 + AA19: 961.7621459960938 + AA20: 888077 + AA21: 79706.90625 + AA22: 392281.65625 + AA23: 486379 + AA24: 2.576596736907959 + AB17: 234.94024658203125 + AB18: 955.3402099609375 + AB19: 939.760986328125 + AB20: 905509 + AB21: 79520.09375 + AB22: 376665.5 + AB23: 473957.46875 + AB24: 2.564022302627564 + AC17: 240.65753173828125 + AC18: 956.7325439453125 + AC19: 962.630126953125 + AC20: 910074 + AC21: 78656.234375 + AC22: 367932.59375 + AC23: 436266 + AC24: 2.563867330551148 + AD17: 244.769287109375 + AD18: 960.8076171875 + AD19: 979.0771484375 + AD20: 864497 + AD21: 82552.578125 + AD22: 335549.75 + AD23: 418456 + AD24: 2.596183776855469 + AE17: 239.0762481689453 + AE18: 959.4432983398438 + AE19: 956.3049926757812 + AE20: 964735 + AE21: 83926.0078125 + AE22: 353485.84375 + AE23: 423065.5 + AE24: 2.636713027954102 + AF17: 237.9830780029297 + AF18: 962.4861450195312 + AF19: 951.9323120117188 + AF20: 951298 + AF21: 85729.078125 + AF22: 333017.28125 + AF23: 375459 + AF24: 2.628951072692871 + AG17: 239.9556121826172 + AG18: 961.7842407226562 + AG19: 959.8224487304688 + AG20: 932617 + AG21: 88160.1953125 + AG22: 324605.5 + AG23: 360714 + AG24: 2.681182622909546 + AH17: 238.0694122314453 + AH18: 955.0843505859375 + AH19: 952.2776489257812 + AH20: 920347 + AH21: 90300.109375 + AH22: 328295.46875 + AH23: 367432.5 + AH24: 2.690957069396973 + S17: 243.58688354492188 + S18: 981.8154296875 + S19: 974.3475341796875 + S20: 671326 + S21: null + S22: 417895.8125 + S23: 507579.5 + S24: 2.540113925933838 + T17: 242.791015625 + T18: 976.6210327148438 + T19: 971.1640625 + T20: 659110 + T21: null + T22: 382527.28125 + T23: 467265 + T24: 2.556509494781494 + U17: 245.53134155273438 + U18: 979.336181640625 + U19: 982.1253662109375 + U20: 657656 + U21: null + U22: 383177.8125 + U23: 480897.5 + U24: 2.54644775390625 + V17: 234.80374145507812 + V18: 966.7130126953125 + V19: 939.2149658203125 + V20: 698109 + V21: 83819.296875 + V22: 376420.0625 + V23: 499018.5 + V24: 2.531107902526856 + W17: 243.1981658935547 + W18: 966.32421875 + W19: 972.7926635742188 + W20: 723673 + W21: 82227.984375 + W22: 368261.6875 + W23: 472664.5 + W24: 2.542474031448364 + X17: 245.06390380859375 + X18: 968.59716796875 + X19: 980.255615234375 + X20: 742352 + X21: 80773.09375 + X22: 363038.25 + X23: 469867.5 + X24: 2.551462650299072 + Y17: 239.26527404785156 + Y18: 962.3310546875 + Y19: 957.0610961914062 + Y20: 773681 + Y21: 79567.2265625 + Y22: 368659 + Y23: 452536.5 + Y24: 2.574562549591064 + Z17: 240.69418334960938 + Z18: 968.2215576171875 + Z19: 962.7767333984375 + Z20: 850985 + Z21: 79070.2421875 + Z22: 380063.65625 + Z23: 462964 + Z24: 2.564170598983765 + diff --git a/data/data_sample1_relative.xlsx b/data/data_sample1_relative.xlsx old mode 100755 new mode 100644 diff --git a/data/data_sample1_relative.yaml b/data/data_sample1_relative.yaml new file mode 100644 index 0000000..2d81130 --- /dev/null +++ b/data/data_sample1_relative.yaml @@ -0,0 +1,130 @@ +data_sample: + AA17: 240.44053649902344 + AA18: 965.4638671875 + AA19: 961.7621459960938 + AA20: 888077 + AA21: 79706.90625 + AA22: 392281.65625 + AA23: 486379 + AA24: 2.576596736907959 + AB17: 234.94024658203125 + AB18: 955.3402099609375 + AB19: 939.760986328125 + AB20: 905509 + AB21: 79520.09375 + AB22: 376665.5 + AB23: 473957.46875 + AB24: 2.564022302627564 + AC17: 240.65753173828125 + AC18: 956.7325439453125 + AC19: 962.630126953125 + AC20: 910074 + AC21: 78656.234375 + AC22: 367932.59375 + AC23: 436266 + AC24: 2.563867330551148 + AD17: 244.769287109375 + AD18: 960.8076171875 + AD19: 979.0771484375 + AD20: 864497 + AD21: 82552.578125 + AD22: 335549.75 + AD23: 418456 + AD24: 2.596183776855469 + AE17: 239.0762481689453 + AE18: 959.4432983398438 + AE19: 956.3049926757812 + AE20: 964735 + AE21: 83926.0078125 + AE22: 353485.84375 + AE23: 423065.5 + AE24: 2.636713027954102 + AF17: 237.9830780029297 + AF18: 962.4861450195312 + AF19: 951.9323120117188 + AF20: 951298 + AF21: 85729.078125 + AF22: 333017.28125 + AF23: 375459 + AF24: 2.628951072692871 + AG17: 239.9556121826172 + AG18: 961.7842407226562 + AG19: 959.8224487304688 + AG20: 932617 + AG21: 88160.1953125 + AG22: 324605.5 + AG23: 360714 + AG24: 2.681182622909546 + AH17: 238.0694122314453 + AH18: 955.0843505859375 + AH19: 952.2776489257812 + AH20: 920347 + AH21: 90300.109375 + AH22: 328295.46875 + AH23: 367432.5 + AH24: 2.690957069396973 + S17: 243.58688354492188 + S18: 981.8154296875 + S19: 974.3475341796875 + S20: 671326 + S21: null + S22: 417895.8125 + S23: 507579.5 + S24: 2.540113925933838 + T17: 242.791015625 + T18: 976.6210327148438 + T19: 971.1640625 + T20: 659110 + T21: null + T22: 382527.28125 + T23: 467265 + T24: 2.556509494781494 + U17: 245.53134155273438 + U18: 979.336181640625 + U19: 982.1253662109375 + U20: 657656 + U21: null + U22: 383177.8125 + U23: 480897.5 + U24: 2.54644775390625 + V17: 234.80374145507812 + V18: 966.7130126953125 + V19: 939.2149658203125 + V20: 698109 + V21: 83819.296875 + V22: 376420.0625 + V23: 499018.5 + V24: 2.531107902526856 + W17: 243.1981658935547 + W18: 966.32421875 + W19: 972.7926635742188 + W20: 723673 + W21: 82227.984375 + W22: 368261.6875 + W23: 472664.5 + W24: 2.542474031448364 + X17: 245.06390380859375 + X18: 968.59716796875 + X19: 980.255615234375 + X20: 742352 + X21: 80773.09375 + X22: 363038.25 + X23: 469867.5 + X24: 2.551462650299072 + Y17: 239.26527404785156 + Y18: 962.3310546875 + Y19: 957.0610961914062 + Y20: 773681 + Y21: 79567.2265625 + Y22: 368659 + Y23: 452536.5 + Y24: 2.574562549591064 + Z17: 240.69418334960938 + Z18: 968.2215576171875 + Z19: 962.7767333984375 + Z20: 850985 + Z21: 79070.2421875 + Z22: 380063.65625 + Z23: 462964 + Z24: 2.564170598983765 + diff --git a/data/do_stone.py b/data/do_stone.py new file mode 100755 index 0000000..05f1aaa --- /dev/null +++ b/data/do_stone.py @@ -0,0 +1,27 @@ +import os +import subprocess + + + +for fname in os.listdir(): + if fname.endswith('xlsx'): + + # Without parser enabled + destname = "{}.yaml".format(os.path.splitext(fname)[0]) + if os.path.exists(destname): + print("{} nothing to do".format(destname)) + else: + print('Doing {}'.format(destname)) + with open(destname, 'wt') as outfile: + subprocess.run(['xltoy','collect',fname,'--yaml'], stdout=outfile) + + + # Parser enabled + destname = "{}.parsed.yaml".format(os.path.splitext(fname)[0]) + if os.path.exists(destname): + print("{} nothing to do".format(destname)) + else: + print('Doing {}'.format(destname)) + with open(destname, 'wt') as outfile: + subprocess.run(['xltoy','collect',fname,'--parsed','--yaml'], stdout=outfile) + diff --git a/data/gla_model.parsed.yaml b/data/gla_model.parsed.yaml new file mode 100644 index 0000000..24b1a1a --- /dev/null +++ b/data/gla_model.parsed.yaml @@ -0,0 +1,7 @@ +Foglio1: + V1: =(V1[T-2]-V1[T-7])*V1[T-6] + V2: =(V2[T-2]-V2[T-7])*V2[T-6]+0.02*X2[T]+0.003*X1[T-2] + V3: =(V3[T-2]-V3[T-7])*V3[T-6]+V3[T-5]*max(V1[T],V2[T]) + X1: 1 + X2: 1 + diff --git a/data/gla_model.xlsx b/data/gla_model.xlsx old mode 100755 new mode 100644 diff --git a/data/gla_model.yaml b/data/gla_model.yaml new file mode 100644 index 0000000..49e0bf7 --- /dev/null +++ b/data/gla_model.yaml @@ -0,0 +1,7 @@ +Foglio1: + V1: =(D8-D$3)*D$4 + V2: =(E8-E$3)*E$4+0.02*B10+0.003*C8 + V3: =(F8-F$3)*F$4+$F$5*MAX(D10,E10) + X1: 1 + X2: 1 + diff --git a/data/gla_multimodel.parsed.yaml b/data/gla_multimodel.parsed.yaml new file mode 100644 index 0000000..20d80c1 --- /dev/null +++ b/data/gla_multimodel.parsed.yaml @@ -0,0 +1,11 @@ +Foglio1: + V1: =(V1[T-2]-V1[T-7])*V1[T-6] + V2: =(V2[T-2]-V2[T-7])*V2[T-6]+0.02*X2[T]+0.003*X1[T-2] + V3: =(V3[T-2]-V3[T-7])*V3[T-6]+V3[T-5]*max(V1[T],V2[T]) + X1: 1 + X2: 1 +Foglio2: + K1: =X2[T]+0.36330198619804854 + K2: =K1[T]+X2[T] + X2: =1 + diff --git a/data/gla_multimodel.xlsx b/data/gla_multimodel.xlsx old mode 100755 new mode 100644 diff --git a/data/gla_multimodel.yaml b/data/gla_multimodel.yaml new file mode 100644 index 0000000..3dce358 --- /dev/null +++ b/data/gla_multimodel.yaml @@ -0,0 +1,11 @@ +Foglio1: + V1: =(D8-D$3)*D$4 + V2: =(E8-E$3)*E$4+0.02*B10+0.003*C8 + V3: =(F8-F$3)*F$4+$F$5*MAX(D10,E10) + X1: 1 + X2: 1 +Foglio2: + K1: =B8+Foglio1!F8 + K2: =D8+B8 + X2: =Foglio1!B8 + diff --git a/data/simple_3.parsed.yaml b/data/simple_3.parsed.yaml new file mode 100644 index 0000000..d902d32 --- /dev/null +++ b/data/simple_3.parsed.yaml @@ -0,0 +1,7 @@ +Foglio1: + V1: =(V1[T-2]-V1[T-7])*V1[T-6] + V2: =(V2[T-2]-V2[T-7])*V2[T-6]+0.02*X2[T]+0.003*X1[T-2] + V3: =(F8-F$3)*F$4+$F$5*MAX(D10,E10)+0*+LEN(V3[T-3]) + X1: 1 + X2: 1 + diff --git a/data/simple_3.xlsx b/data/simple_3.xlsx old mode 100755 new mode 100644 diff --git a/data/simple_3.yaml b/data/simple_3.yaml new file mode 100644 index 0000000..9eb054b --- /dev/null +++ b/data/simple_3.yaml @@ -0,0 +1,7 @@ +Foglio1: + V1: =(D8-D$3)*D$4 + V2: =(E8-E$3)*E$4+0.02*B10+0.003*C8 + V3: =(F8-F$3)*F$4+$F$5*MAX(D10,E10)+0*+LEN(F7) + X1: 1 + X2: 1 + diff --git a/data/upload_HybridModel.parsed.yaml b/data/upload_HybridModel.parsed.yaml new file mode 100644 index 0000000..f4c02ed --- /dev/null +++ b/data/upload_HybridModel.parsed.yaml @@ -0,0 +1,6 @@ +Sheet1: + S: =S[T-1]+S[T-1]*(rf[T]-rf[T-5])*S[T-9]+S[T-1]*rf[T-6]*VS[T-1]*sqrt(S[T-9]) + VS: =rf[T-4] + Vrf: =rf[T-4] + rf: =rf[T-1]+rf[T-9]*(rf[T-8]-rf[T-1])*S[T-9]+rf[T-7]*Vrf[T-1]*sqrt(S[T-9]) + diff --git a/data/upload_HybridModel.xlsx b/data/upload_HybridModel.xlsx old mode 100755 new mode 100644 diff --git a/data/upload_HybridModel.yaml b/data/upload_HybridModel.yaml new file mode 100644 index 0000000..2f507e7 --- /dev/null +++ b/data/upload_HybridModel.yaml @@ -0,0 +1,6 @@ +Sheet1: + S: =D14+D14*(E15 -$E$10)*$D$6 + D14*$E$9*F14*SQRT($D$6) + VS: =$E$11 + Vrf: =$E$11 + rf: =E14+$E$6*($E$7-E14)*$D$6+$E$8*G14*SQRT($D$6) + diff --git a/scripts/cmd.py b/scripts/cmd.py index babecd7..41f6287 100644 --- a/scripts/cmd.py +++ b/scripts/cmd.py @@ -32,14 +32,16 @@ def cli(): @click.option('--data', is_flag=True, help='Collect only data, it will ignore formulas') @click.option('-v', '--verbose', count=True, help="verbose output (repeat for increased verbosity)") @click.option('--add_fingerprint', is_flag=True, help='Add metadata under section xltoy') +@click.option('--parsed', is_flag=True, help='Parse formulas and use this version instead of excel syntax') @click.argument('filename') def collect(filename, **kwargs): set_verb(kwargs.get('verbose')) - click.echo('Collect an excel') with timeit("{} collect".format(filename), kwargs.get('timeit')): c = Collector(filename, only_data=kwargs.get('data'), - add_fingerprint=kwargs.get('add_fingerprint')) + parsed=kwargs.get('parsed'), + add_fingerprint=kwargs.get('add_fingerprint'), + ) if kwargs.get('yaml'): print(c.to_yaml()) @@ -50,6 +52,7 @@ def collect(filename, **kwargs): @click.option('--relative', is_flag=True, help='Areas are handled as relative, each starts from row1,col1') @click.option('-v', '--verbose', count=True, help="verbose output (repeat for increased verbosity)") @click.option('--add_fingerprint', is_flag=True, help='Add metadata under section xltoy') +@click.option('--parsed', is_flag=True, help='Parse formulas and use this version instead of excel syntax') @click.argument('filename1') @click.argument('filename2') def diff(filename1, filename2, **kwargs): @@ -58,6 +61,7 @@ def diff(filename1, filename2, **kwargs): d = DiffCollector(filename1,filename2, only_data=kwargs.get('data'), relative=kwargs.get('relative'), + parsed=kwargs.get('parsed'), add_fingerprint=kwargs.get('add_fingerprint')) d.to_yaml() diff --git a/setup.py b/setup.py index 4e257e3..fbb71bc 100644 --- a/setup.py +++ b/setup.py @@ -1,9 +1,12 @@ import setuptools from xltoy import version -with open("README.md", "r") as fh: +with open("README.md", "rt") as fh: long_description = fh.read() +with open("requirements.txt",'rt') as fn: + reqs = fn.read().split() + setuptools.setup( name="xltoy", version=version, @@ -27,6 +30,7 @@ "Intended Audience :: Information Technology", ], python_requires='>=3.6', + install_requires = reqs, entry_points={ 'console_scripts': [ 'xltoy = scripts.cmd:cli' diff --git a/tests/test_collector.py b/tests/test_collector.py index dc38013..7dfb59c 100644 --- a/tests/test_collector.py +++ b/tests/test_collector.py @@ -1,5 +1,6 @@ import xltoy -from xltoy.collector import Collector +from xltoy.collector import Collector, DiffCollector +from xltoy.utils import is_vertical_range import os import pytest @@ -13,6 +14,47 @@ def coll(url): @pytest.mark.collector -@pytest.mark.parametrize("f", [src for src in list(os.walk(base_data_url))[0][2]]) +@pytest.mark.parametrize("f", [src for src in list(os.walk(base_data_url))[0][2] if src.endswith('.xlsx')]) def test_injestion(f): coll(f) + + +@pytest.mark.collector +@pytest.mark.parametrize("rng", [['A1','A2','A3','$A$4','$A5','A$12','$A$13']]) +def test_is_vertical(rng): + assert is_vertical_range(rng) is False + + +@pytest.mark.collector +@pytest.mark.parametrize("rng", [['AA1','AA2','AA3','$AA5','$AA$6']]) +def test_is_horiz(rng): + assert is_vertical_range(rng) is False + + +@pytest.mark.collector +@pytest.mark.parametrize("rng",[['A1','B2','C3']]) +def test_not_valid_range(rng): + with pytest.raises(ValueError): + is_vertical_range(rng) is True + +@pytest.mark.collector +@pytest.mark.parametrize("rng",[]) +def test_empty_valid_range(rng): + with pytest.raises(ValueError): + is_vertical_range(rng) + +@pytest.mark.collector +@pytest.mark.parametrize("f", [src for src in list(os.walk(base_data_url))[0][2] if src.endswith('.xlsx')]) +def test_diff(f): + f = os.path.join(base_data_url, f) + stone_f = os.path.join( + base_data_url, + os.path.splitext(f)[0]+'.yaml') + if os.path.exists(stone_f): + assert not DiffCollector(f, stone_f).diff + + stone_f = os.path.join( + base_data_url, + os.path.splitext(f)[0]+'.parsed.yaml') + if os.path.exists(stone_f): + assert not DiffCollector(f, stone_f, parsed=True).diff diff --git a/tests/test_parser.py b/tests/test_parser.py index 3f313ea..4323d3b 100644 --- a/tests/test_parser.py +++ b/tests/test_parser.py @@ -1,7 +1,8 @@ import pytest -from xltoy.parser import xl_parse - +from xltoy.parser import Parser +from xltoy.utils import split_sheet_coordinates +p = Parser(collector=None) test_set = """ =3*A7+5 @@ -21,15 +22,20 @@ =IF(D7,D7+E7,D7-E7) =1/(1+EXP(LN(1/INPUT_M!$G$62-1)-E$543)) =+IF(A4+B4, 1/(1+EXP(LN(1/INPUT_M!$G$62-1)-E$543)), IF(A4+B4, 1, PRIVFUN(E4))) -=+IF(PARAM!$B$135=0, 1/(1+EXP(LN(1/INPUT_M!$G$62-1)-E$543)), IF(PARAM!$B$135=1,PRIVFUN(FUNCT(INPUT_M!$G$62)+E$543),PRIVFUN((FUNCT(INPUT_M!$G$62)+SQRT(INPUT_M!$G$45)*E$543)/SQRT(1-INPUT_MATRIX!$G$45)))) +=+IF(PARAM!$B$135=0, 1/(1+EXP(LN(1/INPUT_M!$G$62-1)-E$543)), IF(PARAM!$B$135=1,PRIVFUN(FUNCT(INPUT_M!$G$62)+E$543),PRIVFUN((FUNCT(INPUT_M!$G$62)+SQRT(INPUT_M!$G$45)*E$543)/SQRT(1-INPUT_MATRIX!$G$45)))) =K13-MIN(L$10,0)+IF(SHEE1!M20=1,0,MIN(K10,0))+L15+L16 +=+INTERBANK!FS221 """.splitlines() - -print(test_set) - - @pytest.mark.parser @pytest.mark.parametrize("s", filter(None,test_set)) def test_parser(s): - assert xl_parse(s) + assert p.parse(s) + + +@pytest.mark.parser +def test_split_sheet_coordinates(): + assert split_sheet_coordinates('E4') == (None, 'E4') + assert split_sheet_coordinates('MYSH!E4') == ('MYSH','E4') + assert split_sheet_coordinates("'M Y SH'!E4") == ('M Y SH', 'E4') + diff --git a/tutorial.md b/tutorial.md index a8e3ed5..6d84a4c 100644 --- a/tutorial.md +++ b/tutorial.md @@ -47,4 +47,13 @@ In second workbook range was moved to another position ``` -So in relative mode, no difference found \ No newline at end of file +So in relative mode, no difference found + + + +#### case 3: forecastimg model + +This is an example of a common forecasting model that can be well handled by XLtoy. +![xlsample](https://github.com/glaucouri/xltoy/raw/main/img/simple_model.png?raw=true) +Green cells contain actual (or hystorical) values, model in salmon for the first calculated step, +and in yellow dragged cells, the rest of the model. diff --git a/xltoy/__init__.py b/xltoy/__init__.py index 850fe52..5ad5810 100644 --- a/xltoy/__init__.py +++ b/xltoy/__init__.py @@ -1,6 +1,6 @@ import logging from logging import DEBUG, INFO, WARNING, ERROR, CRITICAL, Formatter, getLogger, StreamHandler -__version__ = '0.1.5' +__version__ = '0.1.6' version = __version__ diff --git a/xltoy/collector.py b/xltoy/collector.py index fa20d71..5cd7175 100644 --- a/xltoy/collector.py +++ b/xltoy/collector.py @@ -1,18 +1,25 @@ +from os.path import isfile +from datetime import datetime +from collections import defaultdict from openpyxl import load_workbook from openpyxl.worksheet.cell_range import CellRange +from openpyxl.utils.cell import coordinate_to_tuple from openpyxl.cell import Cell -from collections import defaultdict -from os.path import isfile -from datetime import datetime from . import log, version +from .utils import is_vertical_range +from .parser import Parser import re import yaml import dictdiffer +ANON_PRFX = 'anon_' + def useless_range(rng): """ - Check if rng is valid and usefull range + Check if rng is valid and useful range. + Empty range is not significative. + :param rng: :return: True | False """ @@ -20,24 +27,26 @@ def useless_range(rng): return True - -class Sheet: - pass - class Collector: + """ + Main Collector class + """ label_names = [re.compile('^varname', re.IGNORECASE), re.compile('^label', re.IGNORECASE)] model_names = [re.compile('^model', re.IGNORECASE)] data_names = [re.compile('^data', re.IGNORECASE)] - def __init__(self, url:str, only_data:bool=False, relative:bool=False, add_fingerprint:bool=False): + def __init__(self, url:str, only_data:bool=False, relative:bool=False, add_fingerprint:bool=False, + parsed:bool=False): """ - Main data injestion, we need 2 instances of the sheet: + Data injestion, we need 2 instances of the sheet: - wb_data: with static data - wb: with the formulas :param url: url to an input excel file :param only_data: read only static values + :param parsed: use parsed formaulas instead of excel version + :param add_fingerprint: add some punctual information in the internal representation :param relative: all area are treated as if they starts from Row1 Col1 """ if not isfile(url): @@ -47,11 +56,17 @@ def __init__(self, url:str, only_data:bool=False, relative:bool=False, add_finge self.pseudo = {} self.url = url self.anon_models = {} + self.parsed = parsed self.labels_as_data = True self.models_as_data = only_data log.debug("Labels read as {}".format( 'data' if self.labels_as_data else 'formula')) log.debug("Models read as {}".format('data' if self.models_as_data else 'formula')) + if self.parsed: + log.info("Parser enabled") + + # workbook injestion, we need 2 instances: data and formulas + # if only_data is true each point to the same instance. self.wb_data = load_workbook(filename=self.url, data_only=True) if only_data: @@ -62,31 +77,75 @@ def __init__(self, url:str, only_data:bool=False, relative:bool=False, add_finge self.add_fingerprint = add_fingerprint self.sheetnames = self.wb_data.sheetnames - self.named_ranges = self.wb_data.get_named_ranges() + self.named_ranges = self.wb_data.defined_names.definedName self.set_ranges() - self.models = self.handle_range( - self.model_names, - self.models_as_data) - self.labels = self.handle_range( - self.label_names, - self.labels_as_data) + self.label_names, + self.labels_as_data) + + # Here we have collected only labels, so if parsed is True + # we need a data structure for bind position of formula to his label + self.sheet_is_vertical = {} + if self.parsed: + # models pre-scan, this solves the anonymous models problems. + # we don't know if they are vertical or horizontal + self.models = self.handle_range( + self.model_names, + True) + + self.find_anonymous_models() + + self.pos_to_label = defaultdict(lambda: {}) + for sheet,rng in self.labels.items(): + if sheet not in self.sheet_is_vertical: + is_vert = is_vertical_range(rng.keys()) + self.sheet_is_vertical[sheet] = is_vert + else: + is_vert = self.sheet_is_vertical[sheet] + for coord, label in rng.items(): + row, col = coordinate_to_tuple(coord) + self.pos_to_label[sheet][col if is_vert else row] = label + + self.parser = Parser(collector=self) + + self.models = self.handle_range( + self.model_names, + self.models_as_data) + else: + self.models = self.handle_range( + self.model_names, + self.models_as_data) + + self.find_anonymous_models() self.data = self.handle_range( self.data_names, use_data=True) + self.set_pseudo_excel() + + + + def find_anonymous_models(self): + """ + Models can be defined without labels, so in this cases we must do + more to: + - Assign default labels + - Understand if model is vertical or horizontal - # Check for anonimous models + :return: + """ + # Check for anonymous models for k,v in self.models.items(): if k not in self.labels: - anon_labels = {'anon_{}'.format(x):'anon_{}'.format(x) for x in range(1,len(v))} + self.sheet_is_vertical[k] = is_vertical_range(v) + anon_labels = {f'{x}': f'{ANON_PRFX}{n+1}' for n, x in enumerate(v)} self.anon_models[k] = '{} anon labels assigned'.format(len(anon_labels)) self.labels[k] = anon_labels - self.set_pseudo_excel() + for k,v in self.anon_models.items(): - log.info("Found anonimous model {} : {}".format(k,v)) + log.warning("Found anonymous model {} : {}".format(k,v)) def set_ranges(self): @@ -134,9 +193,12 @@ def set_ranges(self): def to_relative(self, rng_name:str, cell:Cell) -> str: """ - :param rng_name: - :param cell: - :return: + given a range name and a cell inside it, it return che cell position + as RC as relative to the cell 0,0 + + :param rng_name: collection of cells + :param cell: target cell + :return: relative position of the cell. """ ref_cell = self.ranges[rng_name][0] min_row, min_col = ref_cell.row, ref_cell.column @@ -145,10 +207,12 @@ def to_relative(self, rng_name:str, cell:Cell) -> str: def handle_range(self, labels: list, use_data:bool): """ - iter over collected ranges and - :param labels: - :param use_data: - :return: + iter over collected ranges and restructure all data in a confortable nested dict. + coll[] = {..} + + :param labels: list of RE to use to undestand type of range + :param use_data: Use data or formulas. + :return: collection of all collected cells """ coll = {} for lbl in self.ranges: @@ -160,14 +224,42 @@ def handle_range(self, labels: list, use_data:bool): raise NotImplementedError("Range defined on multiple sheets is not handled") sheet_name = sheet_names.pop() if use_data: - coll[sheet_name] = {self.to_relative(lbl,x) if self.relative else x.coordinate:x.value + coll[sheet_name] = {self.to_relative(lbl,x) + if self.relative + else x.coordinate:x.value for x in cells} else: - coll[sheet_name] = {self.to_relative(lbl,x) if self.relative else x.coordinate:self.wb[sheet_name][x.coordinate].value + coll[sheet_name] = {self.to_relative(lbl,x) + if self.relative + else x.coordinate:self.load_formula(sheet_name, x.coordinate, self.wb[sheet_name][x.coordinate].value) for x in cells} break return coll + + def load_formula(self, sheet:str, position:str, s:str) -> str: + """ + In the case we need to handle a formula we need an optional parameter + to activate the parser. *self.parsed* + If parsed is True so we do a further pass for rewrite in pythonic way. + + :param sheet: sheet containing cell + :param coordinate: cell coordinate + :param s: input formula a simple string + :return: s | transliterated version + """ + if (not self.parsed) or (not isinstance(s,str)): + return s + + log.debug('Parsing {} {} {}'.format(sheet, position, s)) + # In case of anonymous sheet we must force + self.parser.set_current(sheet, coordinate_to_tuple(position)) + ret = self.parser.transform(s) + log.debug('Parsed {}'.format(ret)) + return ret + + + def set_pseudo_excel(self): if not self.pseudo: # Remember dict are ordered! @@ -201,12 +293,13 @@ def __init__(self, url): raise FileNotFoundError("File {} does not exists".format(url)) with open(url) as fin: - self.pseudo= yaml.load(fin, Loader=yaml.FullLoader) + self.pseudo = yaml.load(fin, Loader=yaml.FullLoader) class DiffCollector: - def __init__(self, url1, url2, only_data:bool=False, relative:bool=False, add_fingerprint:bool=False): + def __init__(self, url1, url2, only_data:bool=False, relative:bool=False, add_fingerprint:bool=False, + parsed:bool=False): """ workbook differ, given 2 files (excel, yaml or json) it can do intelligent comparison @@ -214,34 +307,40 @@ def __init__(self, url1, url2, only_data:bool=False, relative:bool=False, add_fi :param url2: :param only_data: ignore formulas and compare only values :param relative: all area are treated as if they starts from Row1 Col1 + :param parsed: use parsed formaulas instead of excel version :param add_fingerprint: """ - c1,c2 = [YamlCollector(url) if url.lower().endswith('yaml') else Collector(url,only_data=only_data, relative=relative, add_fingerprint=add_fingerprint) + c1,c2 = [YamlCollector(url) + if url.lower().endswith('yaml') + else Collector(url,only_data=only_data, relative=relative, + add_fingerprint=add_fingerprint, parsed=parsed) for url in [url1,url2]] self.iter_differs = dictdiffer.diff(c1.pseudo,c2.pseudo) - self.diff = {} + self.do_diff() - def to_yaml(self): - if not self.diff: - for kind, mid, sh_cells in self.iter_differs: - if kind not in self.diff: - self.diff[kind] = {} - if isinstance(mid, list): - if len(mid)==1: - mid = mid[0] - else: - raise RuntimeError("{} not understood".format(mid)) - if kind == 'change': - sheet, label = mid.split('.') + def do_diff(self): + self.diff = {} + for kind, mid, sh_cells in self.iter_differs: + if kind not in self.diff: + self.diff[kind] = {} + if isinstance(mid, list): + if len(mid) == 1: + mid = mid[0] + else: + raise RuntimeError("{} not understood".format(mid)) + if kind == 'change': + sheet, label = mid.split('.') + if sheet not in self.diff[kind]: + self.diff[kind][sheet] = {} + self.diff[kind][sheet][label] = ' -> '.join(['{}'.format(x) for x in sh_cells]) + else: + for sheet, cells in sh_cells: if sheet not in self.diff[kind]: self.diff[kind][sheet] = {} - self.diff[kind][sheet][label] = ' -> '.join(['{}'.format(x) for x in sh_cells]) - else: - for sheet, cells in sh_cells: - if sheet not in self.diff[kind]: - self.diff[kind][sheet] = {} - self.diff[kind][sheet] = cells + self.diff[kind][sheet] = cells + + def to_yaml(self): if self.diff: print(yaml.dump(self.diff)) diff --git a/xltoy/parser.py b/xltoy/parser.py index b366927..dbe8a1c 100644 --- a/xltoy/parser.py +++ b/xltoy/parser.py @@ -1,88 +1,140 @@ # inspired to: excelExpr by Paul McGuire +from openpyxl.utils.cell import coordinate_to_tuple +from . import log +from .utils import de_dollar +from pyparsing import ( + CaselessKeyword, Word, alphas, alphanums, nums, Optional, Group, oneOf, Forward, + infixNotation, opAssoc, dblQuotedString, delimitedList, Combine, Literal, QuotedString, ParserElement, + LineEnd, pyparsing_common as ppc) +ParserElement.enablePackrat() + -from pyparsing import ( - CaselessKeyword, - Suppress, - Word, - alphas, - alphanums, - nums, - Optional, - Group, - oneOf, - Forward, - infixNotation, - opAssoc, - dblQuotedString, - delimitedList, - Combine, - Literal, - QuotedString, - ParserElement, - pyparsing_common as ppc, -) -ParserElement.enablePackrat() -EQ, LPAR, RPAR, COLON, COMMA = map(Suppress, "=():,") -EXCL, DOLLAR = map(Literal, "!$") -multOp = oneOf("* /") -addOp = oneOf("+ -") -words = Word(alphas, alphanums + '_') -sheetRef = words | QuotedString("'", escQuote="''") -colRef = Optional(DOLLAR) + Word(alphas, max=2) -rowRef = Optional(DOLLAR) + Word(nums) -cellRef = Combine( - Group(Optional(sheetRef + EXCL)("sheet") + colRef("col") + rowRef("row")) -) - -cellRange = ( - Group(cellRef("start") + COLON + cellRef("end"))("range") - | cellRef - | Word(alphas, alphanums) -)('cells') - -expr = Forward() - -COMPARISON_OP = oneOf("< = > >= <= != <>") -condExpr = expr + Optional(COMPARISON_OP + expr) - -ifFunc = ( - CaselessKeyword("if") - + LPAR - + Group(condExpr)("condition") - + COMMA - + Group(expr)("if_true") - + COMMA - + Group(expr)("if_false") - + RPAR -) - - -def stat_function(name): - return Group(CaselessKeyword(name) + Group(LPAR + delimitedList(expr) + RPAR)) - - -sumFunc = stat_function("sum") -minFunc = stat_function("min") -maxFunc = stat_function("max") -aveFunc = stat_function("ave") -unknowFunc = words + Group(LPAR + expr + RPAR) - -funcCall = ifFunc | sumFunc | minFunc | maxFunc | aveFunc #| unknowFunc - -numericLiteral = ppc.number -operand = numericLiteral | funcCall | cellRange | cellRef -arithExpr = infixNotation( - operand, [(multOp, 2, opAssoc.LEFT), - ( addOp, 2, opAssoc.LEFT)] -) - -textOperand = dblQuotedString | cellRef -textExpr = infixNotation(textOperand, [("&", 2, opAssoc.LEFT),]) - -expr <<= Optional(addOp)+(arithExpr | textExpr) - -bnf = EQ+expr -xl_parse = bnf.parseString \ No newline at end of file +class Parser: + def __init__(self, collector=None): + self.current_sheet = None + self.collector = collector + EQ, LPAR, RPAR, COLON, COMMA, EXCL, DOLLAR = map(Literal, "=():,!$") + multOp = oneOf("* /") + addOp = oneOf("+ -") + words = Word(alphas, alphanums + '_') + sheetRef = words | QuotedString("'", escQuote="''") + colRef = Optional(DOLLAR) + Word(alphas, max=2) + rowRef = Optional(DOLLAR) + Word(nums) + + cellRef = Combine( + Group(Optional(sheetRef("sheet") + EXCL) + colRef("col") + rowRef("row")) + ).setParseAction(self.cell_action) + + cellRange = ( + Group(cellRef("start") + COLON + cellRef("end"))("range") + | cellRef + | Word(alphas, alphanums) + )('cells') + + expr = Forward() + + COMPARISON_OP = oneOf("< = > >= <= != <>") + condExpr = expr + Optional(COMPARISON_OP + expr) + + ifFunc = ( + CaselessKeyword("if") + + LPAR + + Group(condExpr)("condition") + + COMMA + + Group(expr)("if_true") + + COMMA + + Group(expr)("if_false") + + RPAR + ) + + def stat_function(name): + return Group(CaselessKeyword(name) + Group(LPAR + delimitedList(expr, combine=True) + RPAR)) + + sumFunc = stat_function("sum") + minFunc = stat_function("min") + maxFunc = stat_function("max") + aveFunc = stat_function("ave") + sqrFunc = stat_function("sqrt") + unknowFunc = words + Group(LPAR + expr + RPAR) + + funcCall = ifFunc | sumFunc | minFunc | maxFunc | aveFunc | sqrFunc | unknowFunc + + numericLiteral = ppc.number + operand = numericLiteral | funcCall | cellRange | cellRef + arithExpr = infixNotation( + operand, [(multOp, 2, opAssoc.LEFT), + (addOp, 2, opAssoc.LEFT)], + lpar=LPAR, rpar=RPAR + ) + + textOperand = dblQuotedString | cellRef + textExpr = infixNotation(textOperand, [("&", 2, opAssoc.LEFT), ]) + + atom = (arithExpr | textExpr) + expr << Optional(addOp) + atom + bnf = Optional(EQ|addOp) + expr + LineEnd() + self.bnf = bnf + + def transform(self, *args, **kwargs): + """ + a simple hook to transformString function from pyparsing + + :param args: delegated to child function + :param kwargs: delegated to child function + :return: delegated to child function + """ + return self.bnf.transformString(*args, **kwargs) + + def parse(self, *args, **kwargs): + return self.bnf.parseString(*args, **kwargs) + + def set_current(self, sheet, coordinate): + self.current_sheet = sheet + self.current_row, self.current_col = coordinate + self.current_pos_to_label = self.collector.pos_to_label[sheet] + self.current_sheet_is_vertical = self.collector.sheet_is_vertical[sheet] + + def cell_action(self, tok): + """ + over all token occurrence we do transliteration action! + + tok is a cell reference, in these cases we can transliterate syntax in a + python way. + Depending on flag self.current_sheet_is_vertical we choose the right + direction for the time stepper + + :param tok: exel token like E5, $AA$12 + :return: transliterated syntax + """ + if self.collector is not None: + if not self.current_sheet: + raise ValueError("ggg") + + tok = tok[0] + if '!' in tok: + # maybe from other sheet! + sheet, f_tok = tok.split('!') + if sheet != self.current_sheet: + log.warning(f"In sheet {self.current_sheet} cell {tok} found input from external sheet. Treated as exogenous value") + val = self.collector.wb_data[sheet][f_tok].value + return val + + tok = de_dollar(tok) + + row, col = coordinate_to_tuple(tok) + pos = col if self.current_sheet_is_vertical else row + if pos in self.current_pos_to_label: + label = self.current_pos_to_label[pos] + else: + val = self.collector.wb_data[self.current_sheet][tok].value + log.info("Found params sheet {} cell {}:{}".format(self.current_sheet,tok,val)) + return val + + delta_time = row-self.current_row if self.current_sheet_is_vertical else col-self.current_col + if delta_time: + return("{}[T{}]".format(label,delta_time)) + else: + return("{}[T]".format(label)) diff --git a/xltoy/utils.py b/xltoy/utils.py index fc98c17..4bfdb21 100644 --- a/xltoy/utils.py +++ b/xltoy/utils.py @@ -1,4 +1,7 @@ from contextlib import contextmanager +from openpyxl.cell.cell import Cell +from openpyxl.utils.cell import coordinate_to_tuple +from openpyxl.workbook.defined_name import SHEETRANGE_RE from time import time @@ -17,3 +20,51 @@ def timeit(description: str, enabled:bool=False) -> None: if enabled: print(f"{description} done in {ellapsed_time:.3f} s.") + + +def de_dollar(s:str)-> str: + """ + Remove useless dollars + + :param s: string representing a coordinate + :return: same cell cleanup + + de_dollar('$AA$4') -> 'AA4 + + """ + return s.replace('$','') + +def is_vertical_range(rng:list)->bool: + """ + + :param rng: + :return: + """ + rows_cols = [coordinate_to_tuple(de_dollar(x)) for x in rng] + rows = set((x[0] for x in rows_cols)) + cols = set((x[1] for x in rows_cols)) + if len(rows)==1: + if len(cols)>1: + return True + else: + raise ValueError("Cannot recognize if {} is vertical model range") + else: + # We assume it was >1 + if len(cols) == 1: + return False + else: + raise ValueError("Cannot recognize if {} is horizontal model range") + + +def split_sheet_coordinates(s): + """ + + :param s: + :return: + """ + match = SHEETRANGE_RE.match(s) + if match is None: + return None, s + else: + match = match.groupdict() + return match['quoted'] or match['notquoted'] , match['cells']