Inconsistence between timeseries data shape and variable name with model Output

[jinningwang]

Is your feature request related to a problem? Please describe.
When model Output exist, dae.ts.xy has only desired variables but dae.xy_name still has all variables name. This might be confusing when working with output data.

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Is it an expected behavior to have all variables name but part of the variable data? If not, I can help with adjusting variables name.

[cuihantao]

Right! An overhaul may be needed to make Output behaviors more consistent.

Currently, dae.xy_name stores all the variable names, and dae.ts.xy stores the selected output values.

What's your proposed solution here? Need to make sure the plot and output writing functions work as expected.

[jinningwang]

After some local tests, the output writing function and plotting function work as expected when there are Output devices.
Additionally, there are dae.x_name_output and dae.y_name_output to store the reduced timeseries data.

There are two things I feel worth to do, as they 1) allow better interoperation with AGVis, 2) allow better usage for long time and large size TDS.

1. Add default function parameter TDS.run(from_csv=None)

from_csv is taken care in system.load(). Add an option to use it with TDS.run() is more intuitive and convenient.

andes/andes/routines/tds.py

Lines 344 to 346 in 5ab784b

	# load from csv is provided
	if self.from_csv is not None:
	self.data_csv = self._load_csv(self.from_csv)

2. Add option to complete the input CSV

Auto-complete data_csv when Output info matches with dae timeseries data.

andes/andes/routines/tds.py

Lines 533 to 545 in 5ab784b

	def _csv_step(self):
	"""
	Fetch data for the next step from ``data_csv``.
	"""
	system = self.system
	if self.data_csv is not None:
	system.dae.x[:] = self.data_csv[self.k_csv, 1:system.dae.n + 1]
	system.dae.y[:] = self.data_csv[self.k_csv, system.dae.n + 1:system.dae.n + system.dae.m + 1]
	system.vars_to_models()
	self.converged = True
	return self.converged

[cuihantao]

Yeah, what I meant is that, if you propose a change, you need to check the plot and output writing.

You are right, we already have x_name_outpu, etc. dae.ts.xy is consistent with dae.x_name_output and dae.y_name_output. There is a need for dae.x_name to stay; when one plots with plot(BaseVar), the address indexes into the full variable name list, i.e., dae.x_name_tex and dae.y_name_tex.

[cuihantao]

I made a change to let TDS.run() take from_csv.

I don't think I understand what you mean by autocompleting the input CSV. Explain?

[jinningwang]

When there is model Output, the case cannot replay the generated CSV.

(ams) jinningwang@WangMBA data % andes run npcc2.xlsx -r tds --from-csv npcc2_out.csv --no-output

    _           _         | Version 1.9.1.post24+g7a87ad5d
   /_\  _ _  __| |___ ___ | Python 3.9.18 on Darwin, 04/03/2024 07:32:20 AM
  / _ \| ' \/ _` / -_|_-< | 
 /_/ \_\_||_\__,_\___/__/ | This program comes with ABSOLUTELY NO WARRANTY.

Working directory: "/Users/jinningwang/Documents/work/psal/src/notes/lmp/data"
> Loaded config from file "/Users/jinningwang/.andes/andes.rc"
> Loaded generated Python code in "/Users/jinningwang/.andes/pycode".
Parsing input file "npcc2.xlsx"...
Input file parsed in 0.5067 seconds.
System internal structure set up in 0.0254 seconds.
-> System connectivity check results:
  No islanded bus detected.
  System is interconnected.
  Each island has a slack bus correctly defined and enabled.

-> Power flow calculation
           Numba: Off
   Sparse solver: KLU
 Solution method: NR method
Power flow initialized in 0.0023 seconds.
0: |F(x)| = 0.004215993603
1: |F(x)| = 3.030635298e-08
Converged in 2 iterations in 0.0017 seconds.
CSV data contains fewer variables than required.
Check if the CSV data file is generated from the test case.
GENCLS (vf range) out of typical lower limit.

   idx     | values | limit
-----------+--------+------
 GENCLS_3  | 0.990  | 1    
 GENCLS_4  | 0.926  | 1    
 GENCLS_16 | 0.933  | 1    


Traceback (most recent call last):
  File "/Users/jinningwang/Documents/work/mambaforge/envs/ams/bin/andes", line 33, in <module>
    sys.exit(load_entry_point('andes', 'console_scripts', 'andes')())
  File "/Users/jinningwang/Documents/work/andes/andes/cli.py", line 238, in main
    return func(cli=True, **vars(args))
  File "/Users/jinningwang/Documents/work/andes/andes/main.py", line 615, in run
    system = run_case(cases[0], codegen=codegen, **kwargs)
  File "/Users/jinningwang/Documents/work/andes/andes/main.py", line 407, in run_case
    system.__dict__[routine_cli[r.lower()]].run(**kwargs)
  File "/Users/jinningwang/Documents/work/andes/andes/routines/tds.py", line 353, in run
    self.init()
  File "/Users/jinningwang/Documents/work/andes/andes/routines/tds.py", line 240, in init
    system.dae.x[:] = self.data_csv[0, 1:system.dae.n + 1]
ValueError: could not broadcast input array from shape (140,) into shape (662,)

npcc1 is the normal case with no Output, and npcc2 is the trouble case with Output.

npcc1_out.csv
npcc1.xlsx
npcc2_out.csv
npcc2.xlsx

[cuihantao]

I see! Feel free to come up with a solution and add a test. Not sure if there’s a deeper issue to it. The current program might have all variables access the full dae.x and dae.y arrays. Get Outlook for iOS<https://aka.ms/o0ukef>

________________________________ From: Jinning Wang ***@***.***> Sent: Wednesday, April 3, 2024 6:37:37 AM To: CURENT/andes ***@***.***> Cc: Hantao Cui ***@***.***>; Comment ***@***.***> Subject: Re: [CURENT/andes] Inconsistence between timeseries data shape and variable name with model Output (Issue #526) When there is model Output, the case cannot replay the generated CSV. (ams) ***@***.*** data % andes run npcc2.xlsx -r tds --from-csv npcc2_out.csv --no-output _ _ | Version 1.9.1.post24+g7a87ad5d /_\ _ _ __| |___ ___ | Python 3.9.18 on Darwin, 04/03/2024 07:32:20 AM / _ \| ' \/ _` / -_|_-< | /_/ \_\_||_\__,_\___/__/ | This program comes with ABSOLUTELY NO WARRANTY. Working directory: "/Users/jinningwang/Documents/work/psal/src/notes/lmp/data"

Loaded config from file "/Users/jinningwang/.andes/andes.rc" Loaded generated Python code in "/Users/jinningwang/.andes/pycode".

Parsing input file "npcc2.xlsx"... Input file parsed in 0.5067 seconds. System internal structure set up in 0.0254 seconds. -> System connectivity check results: No islanded bus detected. System is interconnected. Each island has a slack bus correctly defined and enabled. -> Power flow calculation Numba: Off Sparse solver: KLU Solution method: NR method Power flow initialized in 0.0023 seconds. 0: |F(x)| = 0.004215993603 1: |F(x)| = 3.030635298e-08 Converged in 2 iterations in 0.0017 seconds. CSV data contains fewer variables than required. Check if the CSV data file is generated from the test case. GENCLS (vf range) out of typical lower limit. idx | values | limit -----------+--------+------ GENCLS_3 | 0.990 | 1 GENCLS_4 | 0.926 | 1 GENCLS_16 | 0.933 | 1 Traceback (most recent call last): File "/Users/jinningwang/Documents/work/mambaforge/envs/ams/bin/andes", line 33, in <module> sys.exit(load_entry_point('andes', 'console_scripts', 'andes')()) File "/Users/jinningwang/Documents/work/andes/andes/cli.py", line 238, in main return func(cli=True, **vars(args)) File "/Users/jinningwang/Documents/work/andes/andes/main.py", line 615, in run system = run_case(cases[0], codegen=codegen, **kwargs) File "/Users/jinningwang/Documents/work/andes/andes/main.py", line 407, in run_case system.__dict__[routine_cli[r.lower()]].run(**kwargs) File "/Users/jinningwang/Documents/work/andes/andes/routines/tds.py", line 353, in run self.init() File "/Users/jinningwang/Documents/work/andes/andes/routines/tds.py", line 240, in init system.dae.x[:] = self.data_csv[0, 1:system.dae.n + 1] ValueError: could not broadcast input array from shape (140,) into shape (662,) npcc1 is the normal case with no Output, and npcc2 is the trouble case with Output. npcc1_out.csv<https://github.com/CURENT/andes/files/14851482/npcc1_out.csv> npcc1.xlsx<https://github.com/CURENT/andes/files/14851483/npcc1.xlsx> npcc2_out.csv<https://github.com/CURENT/andes/files/14851484/npcc2_out.csv> npcc2.xlsx<https://github.com/CURENT/andes/files/14851485/npcc2.xlsx> — Reply to this email directly, view it on GitHub<#526 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABSNZA5KDC2YHFW5S2352OTY3PSYDAVCNFSM6AAAAABFTVEFSCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZUGM2DAOJRGU>. You are receiving this because you commented.Message ID: ***@***.***>