How to Use -s/-n/-v property with an attribute with spaces

[barny]

Discussed in #32

^{Originally posted by pablogarijo March 2, 2023}
How can I use the -s/-n/-v property with an attribute with spaces (F.E "Planned SW Release Version" or 'Satisfied By Architecture Element')?

I am getting the next error:

dcterms:title,Planned SW Release Version,dcterms:identifier,r
^
Expected one of:

• COMMA
• LBRACE
• DOT
• COLON

I have tried introducing the data with "", with '' and with both at the same time, but I keep getting errors.

oslcquery -J https://doorsng.grupoantolin.com:9443/ -U -P -p "" -C "SW_DALU" -s dcterms:title -F "SW_DALU_V3.0" -q "((oslc:instanceShape='SW Non-Functional Requirement') || (oslc:instanceShape='SW Functional Requirement'))" -n "Planned SW Release Version" -O result3.csv

this is my query

[barny]

For -s this is probably not very obvious, plus there's an issue with -n/-v

Names of attributes have to be surrounded by ' so they are looked up, and also you have to defeat your commandline method for quoting spaces in values. On Windows this means that for an attribute called a space you specify -s "'a space'"

However the processing for -n/-v is different, primarily because the -n/-v values are used two ways; they are put onto the oslc.select list to ensure values (or not) are retrieved for post-filtering, and this needs them to be surrounded by ' like -s values, BUT then during the postprocessing the -n/-v alues are used literally to match column headings which themselves don't have ', so they never match.

At the moment if you can use explicit -s * you can use -n/-v with attribute names with a space in them just by using " around them like -v "a space". They work with -s * because the * suppresses adding -n/-v names to oslc.select, so they're not formally parsed.

There's a simple hacky fix which is to require ' for -n/v names and strip ' off for post-processing, or a compromise hacky fix which is to detect whether they have ' and if not then add them. The best fix might be to require the ' and parse them like -s does, which will also detect if you misnamed the -n/-v value so I think that's my preferred solution.

[pablogarijo]

It worked with QM but when trying to use it on a Global configuration with RM it gives me the next error:

oslcquery -J https://doorsng.grupoantolin.com:9443/ -U -P -A rm,gc,jts:jts23 -p "JEEP KM NA24 AmbLp" -E "JEEP KM NA24 AmbLp (GC)" -G "JEEP KM NA24 AmbLp MASTER V1.0" -s * -O Result.csv

querying for gc config JEEP KM NA24 AmbLp MASTER V1.0
Loading DN shapes: 100%|███████████████████████████████████████████████████████| 170/170 [01:06<00:00, 2.57 results/s]
Preparing Query
Exception on executing request. URL: https://doorsng.grupoantolin.com:9443/rm/views, 400, <rdf:RDF
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:err="http://jazz.net/xmlns/prod/jazz/foundation/1.0/">
rdf:Description
<err:detailedMessage rdf:datatype="http://www.w3.org/2001/XMLSchema#string"
>java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: com.ibm.db2.jcc.am.SqlException: DB2 SQL Error: SQLCODE=-840, SQLSTATE=54004, SQLERRMC=null, DRIVER=4.26.14</err:detailedMessage>
<err:errorMessage rdf:datatype="http://www.w3.org/2001/XMLSchema#string"
>Bad Request</err:errorMessage>
<err:errorStatus rdf:datatype="http://www.w3.org/2001/XMLSchema#long"
>400</err:errorStatus>
</rdf:Description>
</rdf:RDF>

Traceback (most recent call last):
File "C:\Program Files\Python39\Scripts\oslcquery-script.py", line 33, in
sys.exit(load_entry_point('elmclient==0.11.3', 'console_scripts', 'oslcquery')())
File "C:\Users\pablo.garijo\AppData\Roaming\Python\Python39\site-packages\elmclient\examples\oslcquery.py", line 648, in main
do_oslc_query(sys.argv[1:])
File "C:\Users\pablo.garijo\AppData\Roaming\Python\Python39\site-packages\elmclient\examples\oslcquery.py", line 438, in do_oslc_query
results = queryon.do_complex_query( args.resourcetype, querystring=args.query, searchterms=args.searchterms, select=args.select, isnulls=args.null, isnotnulls=args.value
File "C:\Users\pablo.garijo\AppData\Roaming\Python\Python39\site-packages\elmclient\oslcqueryapi.py", line 160, in do_complex_query
resultstack = self._evaluate_steps(querycapabilityuri,querysteps, select=parsedselect, prefixes=prefixes
File "C:\Users\pablo.garijo\AppData\Roaming\Python\Python39\site-packages\elmclient\oslcqueryapi.py", line 281, in _evaluate_steps
results = self.execute_oslc_query(querycapabilityuri,whereterms=[step], select=select, prefixes=prefixes, orderbys=orderbys, searchterms=searchterms, show_progress=show_progress, maxresults=maxresults, delaybetweenpages=delaybetweenpages, pagesize=pagesize, verbose=verbose)
File "C:\Users\pablo.garijo\AppData\Roaming\Python\Python39\site-packages\elmclient\oslcqueryapi.py", line 377, in execute_oslc_query
results = self._execute_vanilla_oslc_query(querycapabilityuri,query_params1, select=select, prefixes=prefixes, show_progress=show_progress, verbose=verbose, maxresults=maxresults, delaybetweenpages=delaybetweenpages, pagesize=pagesize, intent=intent)
File "C:\Users\pablo.garijo\AppData\Roaming\Python\Python39\site-packages\elmclient\oslcqueryapi.py", line 545, in _execute_vanilla_oslc_query
this_result_xml = self.execute_get_rdf_xml(query_url, params=params, headers=headers, cacheable=False, intent=intent)
File "C:\Users\pablo.garijo\AppData\Roaming\Python\Python39\site-packages\elmclient\httpops.py", line 142, in execute_get_rdf_xml
response = request.execute( **kwargs )
File "C:\Users\pablo.garijo\AppData\Roaming\Python\Python39\site-packages\elmclient\httpops.py", line 296, in execute
return self._execute_request( no_error_log=no_error_log, close=close, **kwargs )
File "C:\Users\pablo.garijo\AppData\Roaming\Python\Python39\site-packages\elmclient\httpops.py", line 305, in _execute_request
result = self._execute_one_request_with_login( no_error_log=no_error_log, close=close, **kwargs)
File "C:\Users\pablo.garijo\AppData\Roaming\Python\Python39\site-packages\elmclient\httpops.py", line 486, in _execute_one_request_with_login
response.raise_for_status()
File "C:\Users\pablo.garijo\AppData\Roaming\Python\Python39\site-packages\requests\models.py", line 941, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://doorsng.grupoantolin.com:9443/rm/views?oslc.pageSize=200&oslc.paging=true&oslc.query=true&oslc.select=%2A&projectURL=https%3A%2F%2Fdoorsng.grupoantolin.com%3A9443%2Frm%2Fprocess%2Fproject-areas%2F_H8uFEOi8EeyVs-rf_8FJeA&oslc_config.context=https%3A%2F%2Fdoorsng.grupoantolin.com%3A9443%2Fgc%2Fconfiguration%2F12

[barny]

That error is a new one on me; DB2 is complaining the query is just too big:

54004	The statement has too many table names or too many items in a SELECT or INSERT list.

You've got past authentication because the DN shapes load, also the GC has been resolved to a configuration URL. A similar query (although for a much smaller project) works for me. I've now used oslcquery on very large multi-component queries (ELM on DB2) without hitting this.

I just published version 0.17.0, can you update and try with this?

[barny]

0.17.0 should have resolved the -n/-v problems :-)

[barny]

What version+ifix are you using?

[barny]

Does it work any better if you don't use -s *? Or if you use one of the local configuration for a contribution from RM rather than a GC?

[pablogarijo]

I am currently using the last one 0.17.0. It still gives the same error when trying to query for all the artifacts with artifact type==X inside the global configuration. So I tried to query for all the artifacts with artifact type==X inside each component in the project, with the local configuration for each of them. In most of the cases the query is accepted and the file generates, but I am getting errors in some of them, like this one F.E:

oslcquery -J https://doorsng.grupoantolin.com:9443/ -A rm,gc,jts:jts23 -U -P -p "JEEP KM NA24 AmbLp" -C "JEEP KM NA24 AmbLp - MASTER Technical Specification" -s * -F "JEEP KM NA24 AmbLp - MASTER Technical Specification_V2.0" -O "Archivos/JEEP KM NA24 AmbLp - MASTER Technical Specification.csv" -q "((oslc:instanceShape='SYS Non-Functional Requirement') || (oslc:instanceShape='SYS Functional Requirement') || (oslc:instanceShape='SW Non-Functional Requirement') || (oslc:instanceShape='SW Functional Requirement') || (oslc:instanceShape='SW Low Level Requirement'))"
Exception on retrying request. URL: https://doorsng.grupoantolin.com:9443/rm/cm/stream/_OtxhALKBEe2vAsz6Mdt4cg, 404,
GLC self=<elmclient._rm._RMComponent object at 0x000001CE7FA93DC0> name_or_uri='JEEP KM NA24 AmbLp - MASTER Technical Specification_V2.0'
Loading DN shapes: 100%|███████████████████████████████████████████████████████████| 9/9 [00:00<00:00, 17.61 results/s]
Preparing Query
Identifier s=['oslc:instanceShape']
Identifier s=['oslc:instanceShape']
Identifier s=['oslc:instanceShape']
Identifier s=['oslc:instanceShape']
Identifier s=['oslc:instanceShape']
Query completed in 1 page(s)
Querying : 100%|██████████████████████████████████████████████████████| 322/322 [00:01<00:00, 100.59 results/s]
Query completed in 2 page(s)
Processing : 0%| | 0/455 [00:00<?, ? results/s]Traceback (most recent call last):
File "C:\Program Files\Python39\Scripts\oslcquery-script.py", line 33, in
sys.exit(load_entry_point('elmclient==0.11.3', 'console_scripts', 'oslcquery')())
File "C:\Program Files\Python39\lib\site-packages\elmclient\examples\oslcquery.py", line 648, in main
do_oslc_query(sys.argv[1:])
File "C:\Program Files\Python39\lib\site-packages\elmclient\examples\oslcquery.py", line 438, in do_oslc_query
results = queryon.do_complex_query( args.resourcetype, querystring=args.query, searchterms=args.searchterms, select=args.select, isnulls=args.null, isnotnulls=args.value
File "C:\Program Files\Python39\lib\site-packages\elmclient\oslcqueryapi.py", line 194, in do_complex_query
remappedvalue = self.resolve_uri_to_name(vattr)
File "C:\Program Files\Python39\lib\site-packages\elmclient_project.py", line 268, in resolve_uri_to_name
if ( result := self.app_resolve_uri_to_name(uri) ) is None:
File "C:\Program Files\Python39\lib\site-packages\elmclient_rm.py", line 542, in app_resolve_uri_to_name
result = self.type_name_from_uri(uri)
File "C:\Program Files\Python39\lib\site-packages\elmclient_rm.py", line 567, in type_name_from_uri
raise Exception( f"No type for {uri=}" )
Exception: No type for uri='https://doorsng.grupoantolin.com:9443/rm/types/AT_q6d1QAwNEe2lctQcs4AXBw#_880GU3egGX5EtiCY-NdI9Q'
Processing : 1%|▌ | 4/455 [00:00<00:06, 74.50 results/s]

The -n/ -v problems still exist so I dont know how to get the data.
I am currently getting this error I posted in 5/18 components from a project.

[pablogarijo]

I have tried also without the query but the response is the same

[pablogarijo]

When quering into my STK requirements component I get a different error message:

oslcquery -J https://doorsng.grupoantolin.com:9443/ -A rm,gc,jts:jts23 -U -P -p "JEEP KM NA24 AmbLp" -C "JEEP KM NA24 AmbLp STK" -s * -F "JEEP KM NA24 AmbLp STK_V2.0" -O "Archivos/JEEP KM NA24 AmbLp STK.csv"
Exception on retrying request. URL: https://doorsng.grupoantolin.com:9443/rm/cm/stream/_OtxhALKBEe2vAsz6Mdt4cg, 404,
GLC self=<elmclient._rm._RMComponent object at 0x000001DC5BEE3940> name_or_uri='JEEP KM NA24 AmbLp STK_V2.0'
Loading DN shapes: 100%|█████████████████████████████████████████████████████████| 13/13 [00:00<00:00, 24.31 results/s]
Preparing Query
Traceback (most recent call last):
File "C:\Program Files\Python39\Scripts\oslcquery-script.py", line 33, in
sys.exit(load_entry_point('elmclient==0.11.3', 'console_scripts', 'oslcquery')())
File "C:\Program Files\Python39\lib\site-packages\elmclient\examples\oslcquery.py", line 648, in main
do_oslc_query(sys.argv[1:])
File "C:\Program Files\Python39\lib\site-packages\elmclient\examples\oslcquery.py", line 438, in do_oslc_query
results = queryon.do_complex_query( args.resourcetype, querystring=args.query, searchterms=args.searchterms, select=args.select, isnulls=args.null, isnotnulls=args.value
File "C:\Program Files\Python39\lib\site-packages\elmclient\oslcqueryapi.py", line 160, in do_complex_query
resultstack = self._evaluate_steps(querycapabilityuri,querysteps, select=parsedselect, prefixes=prefixes
File "C:\Program Files\Python39\lib\site-packages\elmclient\oslcqueryapi.py", line 281, in _evaluate_steps
results = self.execute_oslc_query(querycapabilityuri,whereterms=[step], select=select, prefixes=prefixes, orderbys=orderbys, searchterms=searchterms, show_progress=show_progress, maxresults=maxresults, delaybetweenpages=delaybetweenpages, pagesize=pagesize, verbose=verbose)
File "C:\Program Files\Python39\lib\site-packages\elmclient\oslcqueryapi.py", line 377, in execute_oslc_query
results = self._execute_vanilla_oslc_query(querycapabilityuri,query_params1, select=select, prefixes=prefixes, show_progress=show_progress, verbose=verbose, maxresults=maxresults, delaybetweenpages=delaybetweenpages, pagesize=pagesize, intent=intent)
File "C:\Program Files\Python39\lib\site-packages\elmclient\oslcqueryapi.py", line 545, in _execute_vanilla_oslc_query
this_result_xml = self.execute_get_rdf_xml(query_url, params=params, headers=headers, cacheable=False, intent=intent)
File "C:\Program Files\Python39\lib\site-packages\elmclient\httpops.py", line 143, in execute_get_rdf_xml
result = ET.ElementTree(ET.fromstring(response.content))
File "src\lxml\etree.pyx", line 3252, in lxml.etree.fromstring
File "src\lxml\parser.pxi", line 1912, in lxml.etree._parseMemoryDocument
File "src\lxml\parser.pxi", line 1800, in lxml.etree._parseDoc
File "src\lxml\parser.pxi", line 1141, in lxml.etree._BaseParser._parseDoc
File "src\lxml\parser.pxi", line 615, in lxml.etree._ParserContext._handleParseResultDoc
File "src\lxml\parser.pxi", line 725, in lxml.etree._handleParseResult
File "src\lxml\parser.pxi", line 654, in lxml.etree._raiseParseError
File "", line 160
lxml.etree.XMLSyntaxError: EntityRef: expecting ';', line 160, column 95

[barny]

There may be several issues you're hitting, because you have three different failures.

You're trying to query for all artifacts of a particular type across multiple contributions in a GC. Each of these type definitions has a unique local URI in each component. Unfortunately elmclient/oslcquery doesn't currently handle this. If your types all have RDF URIs, and if elmclient/oslcquery discovered the typesystem for each contribution then it may be possible to make this type of query work, but that's a lot of work and not in scope for elmclient which is really only intended as a demonstrator for the APIs, i.e. it's not intended as a comprehensive solution for all needs.

However oslcquery should work in a local configuration, like your last example querying STK requirements component. It looks like something about the XML coming back from DOORS Next is badly formed at line 160 column 95. You can capture a complete trace of the communication by adding -L TRACE to your commandline - this will produce in a subfolder logs a file named for the date/time like elmclient-20230316-09202.log which shows every GET and response including headers and body content - NOTE this will include your username and password so delete after use! The end of the file will show the last XML received, probably the response to the OSLC query, which has some odd content at line 160 column 95.

For this query, does it work any better if you don't use -s *?

FYI your query specification -q "((oslc:instanceShape='SYS Non-Functional Requirement') || (oslc:instanceShape='SYS Functional Requirement') || (oslc:instanceShape='SW Non-Functional Requirement') || (oslc:instanceShape='SW Functional Requirement') || (oslc:instanceShape='SW Low Level Requirement'))"
will require five OSLC Query calls to be made, one for each of the sections of your -q. It will be much quicker/easier and only require one OSLC Query if you use:
-q "oslc:instanceShape in ['SYS Non-Functional Requirement','SYS Functional Requirement','SW Non-Functional Requirement','SW Functional Requirement','SW Low Level Requirement']"

Another FYI - those command lines are long/complicated aren't they. It's possible to put all the query information in an Excel spreadsheet and run all the queries from the spreadsheet, which requires an awful lot less typing. I use this as a basic regression test before each release. so for example I use batchquery tests\tests_702.xlsx to run all 50+ tests. The regression aspect is that you can enter the expected number of results for each query. You could use tests.xlsx as a starting point for your own collection of batch queries.

[barny]

Which version+ifix are you using?

[pablogarijo]

Thank you for your answer,
I am using ifix0x13.
From now on I will use this structure -q "oslc:instanceShape in ['SYS Non-Functional Requirement','SYS Functional Requirement','SW Non-Functional Requirement','SW Functional Requirement','SW Low Level Requirement'].

In my example with STK requirements, it works without a problem if I use the same query without -s *, but then I cannot get the data I want because no attribute information gets displayed.
So now I have 2 options:

• Use the -s * option and generate bigger csv files with all the information. Then parse those documents
• Use the -n/-v option and generate small csv files. (Still gives me problems with spaces in the name of the attribute)

But both options give me problems right now.

And about the length of the queries I use: Yes they are long and complicated, but I generate them with a python file looping through lists with my component names and stream names, so it would be pretty much the same as using the excel file. But thank you for your suggestion.

[barny]

works without -s *

I suspect there's some XML data with a bad (as far as elmclient is concerned) XML entity reference in one of the attributes selected by -s *, almost certainly the Primary Text, and probably created by pasting.

Eliminate most other attributes by using -s "'Primary Text'" - I'd expect you still get the error if the entity ref is in the primary text. if you use the -L TRACE method I described earlier you the response to the last GET will include this XML. It may be easier to find the line/column now there are fewer attributes being returned. Another way would be to divide and conquer by selecting a subset of artifact IDs like -q "dcterms:identifier>1000 and dcterms:identifier<2000" (or whatever range of identifiers gives you roughly half the previous results) and if no error then use the other half range; when you get an error keep dividing that range by two until you reduce to a very small number of identifiers or just one that gives an error. If you can show me the entityref or ideally the complete XHTML of the problem artifact that may help identify the problem.

[pablogarijo]

oslcquery -J https://doorsng.grupoantolin.com:9443/ -A rm,gc,jts:jts23 -U -P -p "JEEP KM NA24 AmbLp" -C "JEEP KM NA24 AmbLp STK" -s "'Primary Text'" -F "JEEP KM NA24 AmbLp STK_V2.0" -O "Archivos/JEEP KM NA24 AmbLp STK.csv" -L TRACE

GLC self=<elmclient._rm._RMComponent object at 0x000001BC6482BB50> name_or_uri='JEEP KM NA24 AmbLp STK_V2.0'
Loading DN shapes: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 13/13 [00:03<00:00, 3.61 results/s]
Preparing Query
Traceback (most recent call last):
File "C:\Program Files\Python39\Scripts\oslcquery-script.py", line 33, in
sys.exit(load_entry_point('elmclient==0.11.3', 'console_scripts', 'oslcquery')())
File "C:\Program Files\Python39\lib\site-packages\elmclient\examples\oslcquery.py", line 648, in main
do_oslc_query(sys.argv[1:])
File "C:\Program Files\Python39\lib\site-packages\elmclient\examples\oslcquery.py", line 438, in do_oslc_query
results = queryon.do_complex_query( args.resourcetype, querystring=args.query, searchterms=args.searchterms, select=args.select, isnulls=args.null, isnotnulls=args.value
File "C:\Program Files\Python39\lib\site-packages\elmclient\oslcqueryapi.py", line 123, in do_complex_query
parsedselect,selectprefixes = self._parse_select(select)
File "C:\Program Files\Python39\lib\site-packages\elmclient\oslcqueryapi.py", line 842, in _parse_select
tree = parser.parse(selectstring)
File "C:\Users\pablo.garijo\AppData\Roaming\Python\Python39\site-packages\lark\lark.py", line 581, in parse
return self.parser.parse(text, start=start, on_error=on_error)
File "C:\Users\pablo.garijo\AppData\Roaming\Python\Python39\site-packages\lark\parser_frontends.py", line 106, in parse
return self.parser.parse(stream, chosen_start, **kw)
File "C:\Users\pablo.garijo\AppData\Roaming\Python\Python39\site-packages\lark\parsers\earley.py", line 297, in parse
to_scan = self._parse(lexer, columns, to_scan, start_symbol)
File "C:\Users\pablo.garijo\AppData\Roaming\Python\Python39\site-packages\lark\parsers\xearley.py", line 144, in _parse
to_scan = scan(i, to_scan)
File "C:\Users\pablo.garijo\AppData\Roaming\Python\Python39\site-packages\lark\parsers\xearley.py", line 118, in scan
raise UnexpectedCharacters(stream, i, text_line, text_column, {item.expect.name for item in to_scan},
lark.exceptions.UnexpectedCharacters: No terminal matches ''' in the current parser context, at line 1 col 1

'Primary Text',dcterms:identifier,rm_nav
^
Expected one of:
* STAR
* URI_REF_ESC
* NAME

[barny]

That looks reminiscent of output from the version before I fixed the single-quote problem with -s. Your pasted text doesn't show the version you're using - it should show as first line printed Version 0.17.0 - so please check you're using that latest version.

A very similar command works for me using 0.17.0:

oslcquery -A rm,gc -p rm_gc_p1 -s "'Primary Text'" -E gcproj -G "gccomp Initial Development"
Version 0.17.0
querying for gc config gccomp Initial Development
Loading DN shapes: 100%|█████████████████████████████████████████████████████████| 46/46 [00:01<00:00, 40.15 results/s]
Preparing Query
Querying : 100%|████████████████████████████████████████████████████| 1478/1478 [00:01<00:00, 675.07 results/s]
Query completed in 8 page(s)
Processing : 100%|███████████████████████████████████████████████████| 1478/1478 [00:00<00:00, 3416.46 results/s]
Processing completed
Query result has 1478 entries
Runtime was 0m 05s

[barny]

The line numbers in your traceback definitely don't match version 0.17.0