Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] [Distribution v1.13.1-rc1+WA982] Fails to extract OL1 / OL2 products generated by PUG #1028

Open
5 of 12 tasks
Woljtek opened this issue Jul 7, 2023 · 11 comments
Open
5 of 12 tasks

[BUG] [Distribution v1.13.1-rc1+WA982] Fails to extract OL1 / OL2 products generated by PUG #1028

Woljtek opened this issue Jul 7, 2023 · 11 comments
Assignees
@pcuq-ads
Labels
bug Something isn't working CCB Issue for CCB checked ops Ticket from ADS operation team priority:minor Set the priority to minor because the production is (almost) not impacted S3 Relative to Sentinel-3 RS Addons to_be_fixed_phase1 Issue to be fixed for RS phase 1

Comments

@Woljtek
Copy link

Woljtek commented Jul 7, 2023

Environment:

  • Delivery tag:
  • Platform: OPS Orange Cloud
  • Configuration:
    Distribution: 1.13.1rc-1 + WA982 (use develop for distribution-worker)
    processing S3-PUG-NRT : 1.13.1-rc1

Current Behavior:
The distribution-worker fails to extract prip-metadata for:

  • S3A_OL_1_EFR____20230527T214047_20230527T222513_20230707T012733_2666_099_186______LN3_D_NT_002.SEN3
  • S3A_OL_1_EFR____20230527T214047_20230527T222513_20230707T012733_2666_099_186______LN3_D_NT_002.SEN3
  • S3B_OL_1_ERR____20230603T212004_20230603T220430_20230706T200927_2666_080_143______LN3_D_NT_002.SEN3
  • S3B_OL_1_EFR____20230514T213951_20230514T222415_20230706T174931_2664_079_243______LN3_D_NT_002.SEN3

Expected Behavior:
Distribution shall be able to publish all products

Steps To Reproduce:
Republish in compression-event the message related to once of the 4 OL1_EFR products. 001_1440_LN3_O_NR_004

Test execution artefacts (i.e. logs, screenshots…)
The current error is the following:

Error on publishing file S3A_OL_1_EFR____20230527T214047_20230527T222513_20230707T012733_2666_099_186______LN3_D_NT_002.SEN3 in PRIP: java.lang.RuntimeException: Error: Number of retries has exceeded while performing saving prip metadata of S3A_OL_1_EFR____20230527T214047_20230527T222513_20230707T012733_2666_099_186______LN3_D_NT_002.SEN3.zip after 4 attempts: ElasticsearchStatusException[Elasticsearch exception [type=mapper_parsing_exception, reason=failed to parse]]; nested: ElasticsearchException[Elasticsearch exception [type=topology_exception, reason=topology_exception: found non-noded intersection between LINESTRING ( -135.579 72.7312, -132.991 73.5004 ) and LINESTRING ( -1.7874499999999784 73.0118, -180.0 73.1260906895317 ) [ (-134.3488795168782, 73.09681386229416, NaN) ]]];
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:176)
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2011)
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1988)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1745)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1702)

Bug Generic Definition of Ready (DoR)

  • The affect version in which the bug has been found is mentioned
  • The context and environment of the bug is detailed
  • The description of the bug is clear and unambiguous
  • The procedure (steps) to reproduce the bug is clearly detailed
  • The tested User Story / features is linked to the bug if available
  • Logs are attached if available
  • A data set attached if available

Bug Generic Definition of Done (DoD)

  • the modification implemented (the solution to fix the bug) is described in the bug.
  • Unit tests & Continuous integration performed - Test results available - Structural Test coverage reported by SONAR
  • Code committed in GIT with right tag or Analysis/Trade Off documentation up-to-date in reference-system-documentation repository
  • Code is compliant with coding rules (SONAR Report as evidence)
  • Acceptance criteria of the related User story are checked and Passed
@Woljtek Woljtek added bug Something isn't working CCB Issue for CCB ops Ticket from ADS operation team priority:blocking Set the priority to blocking because the production is blocked S3 Relative to Sentinel-3 RS Addons WERUM dev Ticket dedicated to WERUM development labels Jul 7, 2023
@Woljtek
Copy link
Author

Woljtek commented Jul 7, 2023 •
edited
Loading

Note: The error message is identical to error message in METADATA => [BUG] [Metadata v1.13.2-rc1] Fails to extract 1 specific SL_1 and 1 SL_2 product generated by PUG #1002

@w-jka
Copy link

w-jka commented Jul 10, 2023

@Woljtek
The WA for #1002 was only implemented in the metadata-extraction so far, as the distribution-worker will only copy the footprint from the extracted product of the metadata-extraction. If the save of the footprint fails on the metadata-extraction end, it won't contain the footprint, and will therefore not run into any issue in the distribution worker - so the current thought process behind the implementation.

I couldn't find the mentioned products in the s3_pug index but only in the s3_l1_ntc index, so I assume that those products might be the ones you mentioned in this issue. Those products do contain a footprint in the index, so the workaround of #1002 was not applied. I could not find any specific discrepancy between the configuration of the mappings for the coordinates so I am a bit lost, as to why ES allows the footprint in the s3_l1_ntc but throws an error on the prip one.

Could you provide the logs of the distribution-worker for further analysis? I could also add the workaround of #1002 again to the distribution-worker but the logs might help to understand what is happening here.

@LAQU156
Copy link

LAQU156 commented Jul 12, 2023

IVV_CCB_2023_w28 : Moved into "Accepted Werum" for correction, Action on @Woljtek side : Please give logs about this issue. Priority blocking, to be fixed phase 1

@LAQU156 LAQU156 added the to_be_fixed_phase1 Issue to be fixed for RS phase 1 label Jul 12, 2023
@LAQU156
Copy link

LAQU156 commented Jul 12, 2023

Werum_CCB_2023_w28 : Moved into "Product Backlog", waiting for logs from @Woljtek

@Woljtek
Copy link
Author

Woljtek commented Jul 12, 2023

@w-jka
I extracted the log around the error message for S3A_OL_1_EFR____20230527T214047_20230527T222513_20230707T012733_2666_099_186______LN3_D_NT_002.SEN3.zip:
Log in LOKI-logs-2023-07-12 16 32 54.txt

@w-jka
Copy link

w-jka commented Jul 13, 2023 •
edited
Loading

@Woljtek
The big issue with this bug is, that the metadata-extraction is working fine, while the distribution-worker is having problems with the same footprints. I could reproduce this error in a small testing environment using the logs you provided yesterday and could identify the reason as to why this behaviour happens:

Test scenario

docker run --rm -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.15.2

This commands starts a single-node elasticsearch instance locally, which is afterwards available on localhost:9200.

curl -X PUT 'localhost:9200/test1' -H 'Content-Type: application/json' -d '{"mappings": { "properties": { "footprint": { "type": "geo_shape" }}}}'

curl -X PUT 'localhost:9200/test2' -H 'Content-Type: application/json' -d '{"mappings": { "properties": { "footprint": { "type": "geo_shape", "strategy" : "recursive", "tree" : "geohash" }}}}'

These two commands create two indices, test1 containing the configuration of the metadata-extraction indices, and test2 containing the configuration of the trip index for the footprint field.

curl -X POST 'localhost:9200/test1/_doc' -H 'Content-Type: application/json' -d '{"footprint":{"orientation":"counterclockwise","coordinates":[[[6.71086,73.2704],[6.59247,74.3056],[6.48148,75.3406],[6.37688,76.3754],[6.27836,77.4103],[6.19037,78.4451],[6.11607,79.4798],[6.05873,80.5141],[6.02545,81.5484],[6.02668,82.5828],[6.07829,83.6169],[6.20854,84.6506],[6.47425,85.684],[7.00462,86.7175],[8.15619,87.7498],[11.5158,88.7788],[46.5131,89.7599],[172.764,89.1264],[178.455,88.1006],[-179.957,87.069],[-179.276,86.0359],[-178.942,85.0024],[-178.776,83.9687],[-178.702,82.9348],[-178.687,81.9007],[-178.71,80.8664],[-178.76,79.8318],[-178.828,78.7973],[-178.911,77.7628],[-179.006,76.7279],[-179.109,75.6929],[-179.219,74.658],[-179.335,73.6228],[-179.456,72.5875],[-179.581,71.552],[-179.71,70.5163],[-179.843,69.4806],[-179.977,68.4444],[179.888,67.408],[179.75,66.3719],[179.61,65.3354],[179.467,64.2987],[179.323,63.2619],[179.179,62.2246],[179.032,61.187],[178.885,60.1492],[178.736,59.1115],[178.586,58.0736],[178.435,57.0352],[178.283,55.9967],[178.13,54.9577],[177.976,53.9186],[177.821,52.8797],[177.666,51.8403],[177.509,50.8003],[177.351,49.7604],[177.192,48.7204],[177.032,47.6801],[176.871,46.639],[176.71,45.5978],[176.548,44.5565],[176.384,43.5153],[176.22,42.4737],[176.054,41.4316],[175.888,40.3894],[175.721,39.347],[175.552,38.3047],[175.383,37.2618],[175.212,36.2185],[175.041,35.1751],[174.868,34.1316],[174.694,33.088],[174.519,32.0443],[174.343,31.0003],[174.165,29.956],[173.987,28.9116],[173.807,27.8672],[173.625,26.8224],[173.443,25.7772],[173.259,24.7322],[173.074,23.6873],[172.887,22.6423],[172.698,21.597],[172.508,20.5513],[172.317,19.5057],[172.124,18.4604],[171.929,17.4149],[171.732,16.3693],[171.534,15.3236],[171.334,14.2779],[171.132,13.2325],[170.927,12.1869],[170.721,11.1417],[170.513,10.0963],[170.303,9.05044],[170.091,8.00506],[169.876,6.96018],[169.658,5.91544],[169.438,4.87067],[169.216,3.82578],[168.991,2.78122],[168.764,1.73708],[168.534,0.693399],[168.3,-0.349973],[168.064,-1.39323],[167.824,-2.43635],[167.582,-3.47922],[167.336,-4.52144],[167.086,-5.56305],[166.833,-6.60426],[166.576,-7.64515],[166.315,-8.68556],[166.05,-9.72536],[165.782,-10.7649],[165.509,-11.8038],[165.231,-12.8416],[164.948,-13.8784],[164.66,-14.9148],[164.367,-15.9509],[164.07,-16.9862],[163.766,-18.0205],[163.457,-19.0537],[163.141,-20.0858],[162.819,-21.1171],[162.491,-22.1476],[162.156,-23.1769],[161.813,-24.2047],[161.463,-25.2311],[161.105,-26.2567],[160.739,-27.2813],[160.364,-28.3042],[159.98,-29.325],[159.586,-30.3445],[159.182,-31.3623],[158.768,-32.3788],[158.342,-33.3932],[157.905,-34.4055],[157.455,-35.4158],[156.992,-36.4238],[156.516,-37.4299],[156.025,-38.4338],[155.519,-39.4344],[154.996,-40.4321],[154.456,-41.4272],[153.898,-42.4199],[153.321,-43.4092],[152.723,-44.3943],[152.103,-45.3755],[151.459,-46.3533],[150.792,-47.3273],[150.615,-47.5785],[153.139,-48.357],[155.738,-49.0783],[158.41,-49.7393],[161.152,-50.337],[163.959,-50.8684],[166.827,-51.3308],[167.795,-51.4691],[168.157,-50.4307],[168.509,-49.3922],[168.851,-48.3536],[169.184,-47.3138],[169.508,-46.2725],[169.824,-45.23],[170.133,-44.1876],[170.436,-43.1455],[170.733,-42.1024],[171.023,-41.0576],[171.307,-40.0122],[171.588,-38.9677],[171.863,-37.9219],[172.135,-36.8758],[172.402,-35.8291],[172.665,-34.7815],[172.925,-33.7339],[173.182,-32.6862],[173.436,-31.6382],[173.687,-30.5892],[173.935,-29.5391],[174.181,-28.4892],[174.425,-27.4398],[174.667,-26.3896],[174.907,-25.3388],[175.146,-24.2875],[175.382,-23.2363],[175.618,-22.185],[175.852,-21.1331],[176.085,-20.0804],[176.317,-19.0273],[176.548,-17.9742],[176.779,-16.9216],[177.009,-15.8695],[177.239,-14.8167],[177.468,-13.7626],[177.696,-12.708],[177.924,-11.654],[178.153,-10.6001],[178.381,-9.54598],[178.61,-8.49199],[178.839,-7.43803],[179.068,-6.38359],[179.297,-5.32853],[179.527,-4.27356],[179.758,-3.21942],[179.99,-2.16511],[-179.778,-1.11069],[-179.545,-0.056098],[-179.311,0.998682],[-179.076,2.05349],[-178.839,3.10765],[-178.601,4.16141],[-178.361,5.21529],[-178.12,6.26976],[-177.878,7.32464],[-177.633,8.37858],[-177.386,9.43121],[-177.137,10.4843],[-176.886,11.5373],[-176.633,12.5903],[-176.377,13.6431],[-176.119,14.6952],[-175.857,15.7467],[-175.592,16.7979],[-175.324,17.8493],[-175.054,18.9005],[-174.779,19.9507],[-174.5,20.9999],[-174.218,22.0491],[-173.931,23.0981],[-173.64,24.147],[-173.345,25.1949],[-173.044,26.2417],[-172.738,27.2879],[-172.426,28.3337],[-172.109,29.3786],[-171.785,30.4227],[-171.455,31.4661],[-171.118,32.5087],[-170.773,33.5504],[-170.42,34.5912],[-170.06,35.631],[-169.691,36.6694],[-169.312,37.7069],[-168.923,38.7434],[-168.524,39.7787],[-168.114,40.8123],[-167.692,41.8445],[-167.257,42.8755],[-166.808,43.9051],[-166.345,44.9331],[-165.866,45.9585],[-165.371,46.9815],[-164.858,48.0032],[-164.325,49.0234],[-163.773,50.0411],[-163.198,51.0556],[-162.599,52.0676],[-161.974,53.077],[-161.321,54.0827],[-160.638,55.0855],[-159.921,56.0842],[-159.169,57.0788],[-158.378,58.0696],[-157.545,59.0555],[-156.665,60.0365],[-155.734,61.0109],[-154.747,61.9788],[-153.698,62.9402],[-152.581,63.8938],[-151.39,64.8393],[-150.116,65.7754],[-148.749,66.6995],[-147.28,67.6121],[-145.698,68.511],[-143.989,69.3948],[-142.14,70.2614],[-140.134,71.1082],[-137.954,71.9326],[-135.579,72.7312],[-132.991,73.5004],[-130.167,74.2361],[-127.086,74.9332],[-123.727,75.586],[-120.074,76.1883],[-116.115,76.7334],[-111.848,77.2138],[-107.282,77.6224],[-102.442,77.9517],[-97.3725,78.195],[-92.1362,78.3468],[-86.8079,78.4034],[-81.4719,78.3636],[-76.2127,78.2285],[-71.1086,78.0015],[-66.2242,77.6871],[-61.6072,77.2921],[-57.2864,76.8239],[-53.273,76.2896],[-49.5656,75.6968],[-46.1556,75.0524],[-43.0265,74.363],[-40.1581,73.6339],[-37.5289,72.8702],[-35.1175,72.0766],[-32.9052,71.2569],[-30.8716,70.4141],[-28.9957,69.551],[-28.5341,69.3248],[-24.0301,70.3461],[-19.0838,71.2437],[-13.6996,71.9993],[-7.91098,72.5942],[-1.78745,73.0118],[4.56514,73.2386],[6.71086,73.2704]]],"type":"polygon"}}'

--> {"_index":"test1","_type":"_doc","_id":"5o19TokBIlO8xbWLCYGK","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":0,"_primary_term":1}

When publishing the footprint on test1 we receive a successful ingestion. This behaviour is identical to the one on the ops env.

curl -X POST 'localhost:9200/test2/_doc' -H 'Content-Type: application/json' -d '{"footprint":{"orientation":"counterclockwise","coordinates":[[[6.71086,73.2704],[6.59247,74.3056],[6.48148,75.3406],[6.37688,76.3754],[6.27836,77.4103],[6.19037,78.4451],[6.11607,79.4798],[6.05873,80.5141],[6.02545,81.5484],[6.02668,82.5828],[6.07829,83.6169],[6.20854,84.6506],[6.47425,85.684],[7.00462,86.7175],[8.15619,87.7498],[11.5158,88.7788],[46.5131,89.7599],[172.764,89.1264],[178.455,88.1006],[-179.957,87.069],[-179.276,86.0359],[-178.942,85.0024],[-178.776,83.9687],[-178.702,82.9348],[-178.687,81.9007],[-178.71,80.8664],[-178.76,79.8318],[-178.828,78.7973],[-178.911,77.7628],[-179.006,76.7279],[-179.109,75.6929],[-179.219,74.658],[-179.335,73.6228],[-179.456,72.5875],[-179.581,71.552],[-179.71,70.5163],[-179.843,69.4806],[-179.977,68.4444],[179.888,67.408],[179.75,66.3719],[179.61,65.3354],[179.467,64.2987],[179.323,63.2619],[179.179,62.2246],[179.032,61.187],[178.885,60.1492],[178.736,59.1115],[178.586,58.0736],[178.435,57.0352],[178.283,55.9967],[178.13,54.9577],[177.976,53.9186],[177.821,52.8797],[177.666,51.8403],[177.509,50.8003],[177.351,49.7604],[177.192,48.7204],[177.032,47.6801],[176.871,46.639],[176.71,45.5978],[176.548,44.5565],[176.384,43.5153],[176.22,42.4737],[176.054,41.4316],[175.888,40.3894],[175.721,39.347],[175.552,38.3047],[175.383,37.2618],[175.212,36.2185],[175.041,35.1751],[174.868,34.1316],[174.694,33.088],[174.519,32.0443],[174.343,31.0003],[174.165,29.956],[173.987,28.9116],[173.807,27.8672],[173.625,26.8224],[173.443,25.7772],[173.259,24.7322],[173.074,23.6873],[172.887,22.6423],[172.698,21.597],[172.508,20.5513],[172.317,19.5057],[172.124,18.4604],[171.929,17.4149],[171.732,16.3693],[171.534,15.3236],[171.334,14.2779],[171.132,13.2325],[170.927,12.1869],[170.721,11.1417],[170.513,10.0963],[170.303,9.05044],[170.091,8.00506],[169.876,6.96018],[169.658,5.91544],[169.438,4.87067],[169.216,3.82578],[168.991,2.78122],[168.764,1.73708],[168.534,0.693399],[168.3,-0.349973],[168.064,-1.39323],[167.824,-2.43635],[167.582,-3.47922],[167.336,-4.52144],[167.086,-5.56305],[166.833,-6.60426],[166.576,-7.64515],[166.315,-8.68556],[166.05,-9.72536],[165.782,-10.7649],[165.509,-11.8038],[165.231,-12.8416],[164.948,-13.8784],[164.66,-14.9148],[164.367,-15.9509],[164.07,-16.9862],[163.766,-18.0205],[163.457,-19.0537],[163.141,-20.0858],[162.819,-21.1171],[162.491,-22.1476],[162.156,-23.1769],[161.813,-24.2047],[161.463,-25.2311],[161.105,-26.2567],[160.739,-27.2813],[160.364,-28.3042],[159.98,-29.325],[159.586,-30.3445],[159.182,-31.3623],[158.768,-32.3788],[158.342,-33.3932],[157.905,-34.4055],[157.455,-35.4158],[156.992,-36.4238],[156.516,-37.4299],[156.025,-38.4338],[155.519,-39.4344],[154.996,-40.4321],[154.456,-41.4272],[153.898,-42.4199],[153.321,-43.4092],[152.723,-44.3943],[152.103,-45.3755],[151.459,-46.3533],[150.792,-47.3273],[150.615,-47.5785],[153.139,-48.357],[155.738,-49.0783],[158.41,-49.7393],[161.152,-50.337],[163.959,-50.8684],[166.827,-51.3308],[167.795,-51.4691],[168.157,-50.4307],[168.509,-49.3922],[168.851,-48.3536],[169.184,-47.3138],[169.508,-46.2725],[169.824,-45.23],[170.133,-44.1876],[170.436,-43.1455],[170.733,-42.1024],[171.023,-41.0576],[171.307,-40.0122],[171.588,-38.9677],[171.863,-37.9219],[172.135,-36.8758],[172.402,-35.8291],[172.665,-34.7815],[172.925,-33.7339],[173.182,-32.6862],[173.436,-31.6382],[173.687,-30.5892],[173.935,-29.5391],[174.181,-28.4892],[174.425,-27.4398],[174.667,-26.3896],[174.907,-25.3388],[175.146,-24.2875],[175.382,-23.2363],[175.618,-22.185],[175.852,-21.1331],[176.085,-20.0804],[176.317,-19.0273],[176.548,-17.9742],[176.779,-16.9216],[177.009,-15.8695],[177.239,-14.8167],[177.468,-13.7626],[177.696,-12.708],[177.924,-11.654],[178.153,-10.6001],[178.381,-9.54598],[178.61,-8.49199],[178.839,-7.43803],[179.068,-6.38359],[179.297,-5.32853],[179.527,-4.27356],[179.758,-3.21942],[179.99,-2.16511],[-179.778,-1.11069],[-179.545,-0.056098],[-179.311,0.998682],[-179.076,2.05349],[-178.839,3.10765],[-178.601,4.16141],[-178.361,5.21529],[-178.12,6.26976],[-177.878,7.32464],[-177.633,8.37858],[-177.386,9.43121],[-177.137,10.4843],[-176.886,11.5373],[-176.633,12.5903],[-176.377,13.6431],[-176.119,14.6952],[-175.857,15.7467],[-175.592,16.7979],[-175.324,17.8493],[-175.054,18.9005],[-174.779,19.9507],[-174.5,20.9999],[-174.218,22.0491],[-173.931,23.0981],[-173.64,24.147],[-173.345,25.1949],[-173.044,26.2417],[-172.738,27.2879],[-172.426,28.3337],[-172.109,29.3786],[-171.785,30.4227],[-171.455,31.4661],[-171.118,32.5087],[-170.773,33.5504],[-170.42,34.5912],[-170.06,35.631],[-169.691,36.6694],[-169.312,37.7069],[-168.923,38.7434],[-168.524,39.7787],[-168.114,40.8123],[-167.692,41.8445],[-167.257,42.8755],[-166.808,43.9051],[-166.345,44.9331],[-165.866,45.9585],[-165.371,46.9815],[-164.858,48.0032],[-164.325,49.0234],[-163.773,50.0411],[-163.198,51.0556],[-162.599,52.0676],[-161.974,53.077],[-161.321,54.0827],[-160.638,55.0855],[-159.921,56.0842],[-159.169,57.0788],[-158.378,58.0696],[-157.545,59.0555],[-156.665,60.0365],[-155.734,61.0109],[-154.747,61.9788],[-153.698,62.9402],[-152.581,63.8938],[-151.39,64.8393],[-150.116,65.7754],[-148.749,66.6995],[-147.28,67.6121],[-145.698,68.511],[-143.989,69.3948],[-142.14,70.2614],[-140.134,71.1082],[-137.954,71.9326],[-135.579,72.7312],[-132.991,73.5004],[-130.167,74.2361],[-127.086,74.9332],[-123.727,75.586],[-120.074,76.1883],[-116.115,76.7334],[-111.848,77.2138],[-107.282,77.6224],[-102.442,77.9517],[-97.3725,78.195],[-92.1362,78.3468],[-86.8079,78.4034],[-81.4719,78.3636],[-76.2127,78.2285],[-71.1086,78.0015],[-66.2242,77.6871],[-61.6072,77.2921],[-57.2864,76.8239],[-53.273,76.2896],[-49.5656,75.6968],[-46.1556,75.0524],[-43.0265,74.363],[-40.1581,73.6339],[-37.5289,72.8702],[-35.1175,72.0766],[-32.9052,71.2569],[-30.8716,70.4141],[-28.9957,69.551],[-28.5341,69.3248],[-24.0301,70.3461],[-19.0838,71.2437],[-13.6996,71.9993],[-7.91098,72.5942],[-1.78745,73.0118],[4.56514,73.2386],[6.71086,73.2704]]],"type":"polygon"}}'

--> {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"failed to parse"}],"type":"mapper_parsing_exception","reason":"failed to parse","caused_by": {"type":"topology_exception","reason":"found non-noded intersection between LINESTRING ( -135.579 72.7312, -132.991 73.5004 ) and LINESTRING ( -1.7874499999999784 73.0118, -180.0 73.1260906895317 ) [ (-134.3488795168782, 73.09681386229416, NaN) ]"}},"status":400}

When publishing the same footprint to test2 we get an error 400, "non-noded intersection". This behaviour is also the same as on the ops env.

What is happening?
The configuration of a strategy or tree property on a geo_shape field is changing the algorithm that is used by ES from a BKD-backed approach to an older algorithm (prefix tree). The prefix tree is deprecated with ES 7 (and removed in ES 8 and OpenSearch) and it is not advised to use it, as it is slower and uses more space. In our case additionally, the prefix tree has problems with footprints that are above the north and south pole.
You can read more about this in the official documentation and blog post:
https://www.elastic.co/guide/en/elasticsearch/reference/7.15/geo-shape.html#geo-shape-mapping-options
https://www.elastic.co/de/blog/bkd-backed-geo-shapes-in-elasticsearch-precision-efficiency-speed

How can we change the algorithm?
The migration can not be performed on the fly. A new index with the correct settings has to be created in order to use the correct algorithm.

@w-jka
Copy link

w-jka commented Jul 13, 2023 •
edited
Loading

Documentation on how to perform a reindex of data from one index to another
https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html

So in order to fix this issue one approach is:

  1. Undeploy distribution-chain
  2. Create a temporary prip-index (ex. prip_tmp)
  3. Reindex all data from prip to temporary prip-index
  4. Delete prip index and create new prip index with fixed configuration
  5. Reindex all data from temporary prip-index to new prip
  6. Deploy distribution-chain
  7. After everything is working as expected, delete temporary prip-index

@w-jka
Copy link

w-jka commented Jul 13, 2023

Candidate for refusal

@pcuq-ads pcuq-ads mentioned this issue Jul 18, 2023
Closed
12 tasks
@pcuq-ads
Copy link

pcuq-ads commented Jul 26, 2023 •
edited by vgava-ads
Loading

Werum_CCB_2023_w30 : to be checked on ADS side :

@pcuq-ads pcuq-ads self-assigned this Jul 26, 2023
@pcuq-ads
Copy link

pcuq-ads commented Jul 26, 2023 •
edited by vgava-ads
Loading

System_CCB_2023-w30 : The RS Core - Elasticsearch Configuration is in line with the proposed fixed since the version 1.5 . The issue is still present in OPS, but a deployment from scratch will not have the issue. Priority reduced to minor with status CHECKED.
See page : https://github.com/COPRS/production-common/blob/release/1.5.0/processing-common/doc/indices.md

@pcuq-ads pcuq-ads added checked priority:minor Set the priority to minor because the production is (almost) not impacted and removed priority:blocking Set the priority to blocking because the production is blocked WERUM dev Ticket dedicated to WERUM development labels Jul 26, 2023
@suberti-ads
Copy link

suberti-ads commented Jul 28, 2023 •
edited
Loading

For information , this issue impacted also OL2 product
for sample S3B_OL_2_LRR____20230603T212004_20230603T220430_20230707T125951_2666_080_143______LN3_D_NT_002

Error on publishing file S3B_OL_2_LRR____20230603T212004_20230603T220430_20230707T125951_2666_080_143______LN3_D_NT_002.SEN3 in PRIP: java.lang.RuntimeException: Error: Number of retries has exceeded while performing saving prip metadata of S3B_OL_2_LRR____20230603T212004_20230603T220430_20230707T125951_2666_080_143______LN3_D_NT_002.SEN3.zip after 4 attempts: ElasticsearchStatusException[Elasticsearch exception [type=mapper_parsing_exception, reason=failed to parse]]; nested: ElasticsearchException[Elasticsearch exception [type=topology_exception, reason=topology_exception: found non-noded intersection between LINESTRING ( -134.351 71.3427, -129.757 72.9269 ) and LINESTRING ( -1.8263900000000035 71.5222, -180.0 71.72491214102453 ) [ (-133.39638594926308, 71.67189015654711, NaN) ]]];
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:176)
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2011)
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1988)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1745)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1702)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1672)
	at org.elasticsearch.client.RestHighLevelClient.index(RestHighLevelClient.java:1029)
	at esa.s1pdgs.cpoc.prip.metadata.PripElasticSearchMetadataRepo.save(PripElasticSearchMetadataRepo.java:101)
	at esa.s1pdgs.cpoc.prip.worker.service.PripPublishingService.lambda$createAndSave$0(PripPublishingService.java:198)
	at esa.s1pdgs.cpoc.common.utils.Retries.performWithRetries(Retries.java:23)
	at esa.s1pdgs.cpoc.prip.worker.service.PripPublishingService.createAndSave(PripPublishingService.java:197)
	at esa.s1pdgs.cpoc.prip.worker.service.PripPublishingService.accept(PripPublishingService.java:105)
	at esa.s1pdgs.cpoc.prip.worker.service.PripPublishingService.accept(PripPublishingService.java:56)
	at org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.invokeConsumer(SimpleFunctionRegistry.java:976)
	at org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.doApply(SimpleFunctionRegistry.java:705)
	at org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.apply(SimpleFunctionRegistry.java:551)
	at org.springframework.cloud.stream.function.PartitionAwareFunctionWrapper.apply(PartitionAwareFunctionWrapper.java:84)
	at org.springframework.cloud.stream.function.FunctionConfiguration$FunctionWrapper.apply(FunctionConfiguration.java:754)
	at org.springframework.cloud.stream.function.FunctionConfiguration$FunctionToDestinationBinder$1.handleMessageInternal(FunctionConfiguration.java:586)
	at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:56)
	at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:115)
	at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:133)
	at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:106)
	at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:72)
	at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:317)
	at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:272)
	at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:187)
	at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:166)
	at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:47)
	at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:109)
	at org.springframework.integration.endpoint.MessageProducerSupport.sendMessage(MessageProducerSupport.java:216)
	at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter.sendMessageIfAny(KafkaMessageDrivenChannelAdapter.java:397)
	at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter.access$300(KafkaMessageDrivenChannelAdapter.java:83)
	at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter$IntegrationRecordMessageListener.onMessage(KafkaMessageDrivenChannelAdapter.java:454)
	at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter$IntegrationRecordMessageListener.onMessage(KafkaMessageDrivenChannelAdapter.java:428)
	at org.springframework.kafka.listener.adapter.RetryingMessageListenerAdapter.lambda$onMessage$0(RetryingMessageListenerAdapter.java:125)
	at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:329)
	at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:255)
	at org.springframework.kafka.listener.adapter.RetryingMessageListenerAdapter.onMessage(RetryingMessageListenerAdapter.java:119)
	at org.springframework.kafka.listener.adapter.RetryingMessageListenerAdapter.onMessage(RetryingMessageListenerAdapter.java:42)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:2629)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeOnMessage(KafkaMessageListenerContainer.java:2609)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:2536)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeWithRecords(KafkaMessageListenerContainer.java:2427)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListener(KafkaMessageListenerContainer.java:2305)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeListener(KafkaMessageListenerContainer.java:1979)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeIfHaveRecords(KafkaMessageListenerContainer.java:1364)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1355)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1247)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.lang.Thread.run(Thread.java:829)
	Suppressed: org.elasticsearch.client.ResponseException: method [PUT], host [http://elasticsearch-processing-es-http.database.svc.cluster.local:9200], URI [/prip/_doc/S3B_OL_2_LRR____20230603T212004_20230603T220430_20230707T125951_2666_080_143______LN3_D_NT_002.SEN3.zip?timeout=1m], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"topology_exception","reason":"topology_exception: found non-noded intersection between LINESTRING ( -134.351 71.3427, -129.757 72.9269 ) and LINESTRING ( -1.8263900000000035 71.5222, -180.0 71.72491214102453 ) [ (-133.39638594926308, 71.67189015654711, NaN) ]"}],"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"topology_exception","reason":"topology_exception: found non-noded intersection between LINESTRING ( -134.351 71.3427, -129.757 72.9269 ) and LINESTRING ( -1.8263900000000035 71.5222, -180.0 71.72491214102453 ) [ (-133.39638594926308, 71.67189015654711, NaN) ]"}},"status":400}
		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:326)
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:296)
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270)
		at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2082)
		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1732)
		... 48 more
Caused by: ElasticsearchException[Elasticsearch exception [type=topology_exception, reason=topology_exception: found non-noded intersection between LINESTRING ( -134.351 71.3427, -129.757 72.9269 ) and LINESTRING ( -1.8263900000000035 71.5222, -180.0 71.72491214102453 ) [ (-133.39638594926308, 71.67189015654711, NaN) ]]]
	at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:485)
	at org.elasticsearch.ElasticsearchException.fromXContent(ElasticsearchException.java:396)
	at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:426)
	at org.elasticsearch.ElasticsearchException.failureFromXContent(ElasticsearchException.java:592)
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:168)
	... 51 more

	at esa.s1pdgs.cpoc.common.utils.Retries.throwRuntimeException(Retries.java:53)
	at esa.s1pdgs.cpoc.common.utils.Retries.performWithRetries(Retries.java:28)
	at esa.s1pdgs.cpoc.prip.worker.service.PripPublishingService.createAndSave(PripPublishingService.java:197)
	at esa.s1pdgs.cpoc.prip.worker.service.PripPublishingService.accept(PripPublishingService.java:105)
	at esa.s1pdgs.cpoc.prip.worker.service.PripPublishingService.accept(PripPublishingService.java:56)
	at org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.invokeConsumer(SimpleFunctionRegistry.java:976)
	at org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.doApply(SimpleFunctionRegistry.java:705)
	at org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.apply(SimpleFunctionRegistry.java:551)
	at org.springframework.cloud.stream.function.PartitionAwareFunctionWrapper.apply(PartitionAwareFunctionWrapper.java:84)
	at org.springframework.cloud.stream.function.FunctionConfiguration$FunctionWrapper.apply(FunctionConfiguration.java:754)
	at org.springframework.cloud.stream.function.FunctionConfiguration$FunctionToDestinationBinder$1.handleMessageInternal(FunctionConfiguration.java:586)
	at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:56)
	at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:115)
	at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:133)
	at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:106)
	at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:72)
	at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:317)
	at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:272)
	at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:187)
	at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:166)
	at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:47)
	at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:109)
	at org.springframework.integration.endpoint.MessageProducerSupport.sendMessage(MessageProducerSupport.java:216)
	at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter.sendMessageIfAny(KafkaMessageDrivenChannelAdapter.java:397)
	at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter.access$300(KafkaMessageDrivenChannelAdapter.java:83)
	at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter$IntegrationRecordMessageListener.onMessage(KafkaMessageDrivenChannelAdapter.java:454)
	at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter$IntegrationRecordMessageListener.onMessage(KafkaMessageDrivenChannelAdapter.java:428)
	at org.springframework.kafka.listener.adapter.RetryingMessageListenerAdapter.lambda$onMessage$0(RetryingMessageListenerAdapter.java:125)
	at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:329)
	at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:255)
	at org.springframework.kafka.listener.adapter.RetryingMessageListenerAdapter.onMessage(RetryingMessageListenerAdapter.java:119)
	at org.springframework.kafka.listener.adapter.RetryingMessageListenerAdapter.onMessage(RetryingMessageListenerAdapter.java:42)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:2629)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeOnMessage(KafkaMessageListenerContainer.java:2609)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:2536)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeWithRecords(KafkaMessageListenerContainer.java:2427)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListener(KafkaMessageListenerContainer.java:2305)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeListener(KafkaMessageListenerContainer.java:1979)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeIfHaveRecords(KafkaMessageListenerContainer.java:1364)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:1355)
	at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:1247)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: ElasticsearchStatusException[Elasticsearch exception [type=mapper_parsing_exception, reason=failed to parse]]; nested: ElasticsearchException[Elasticsearch exception [type=topology_exception, reason=topology_exception: found non-noded intersection between LINESTRING ( -134.351 71.3427, -129.757 72.9269 ) and LINESTRING ( -1.8263900000000035 71.5222, -180.0 71.72491214102453 ) [ (-133.39638594926308, 71.67189015654711, NaN) ]]];
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:176)
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2011)
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1988)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1745)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1702)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1672)
	at org.elasticsearch.client.RestHighLevelClient.index(RestHighLevelClient.java:1029)
	at esa.s1pdgs.cpoc.prip.metadata.PripElasticSearchMetadataRepo.save(PripElasticSearchMetadataRepo.java:101)
	at esa.s1pdgs.cpoc.prip.worker.service.PripPublishingService.lambda$createAndSave$0(PripPublishingService.java:198)
	at esa.s1pdgs.cpoc.common.utils.Retries.performWithRetries(Retries.java:23)
	... 42 more
	Suppressed: org.elasticsearch.client.ResponseException: method [PUT], host [http://elasticsearch-processing-es-http.database.svc.cluster.local:9200], URI [/prip/_doc/S3B_OL_2_LRR____20230603T212004_20230603T220430_20230707T125951_2666_080_143______LN3_D_NT_002.SEN3.zip?timeout=1m], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"topology_exception","reason":"topology_exception: found non-noded intersection between LINESTRING ( -134.351 71.3427, -129.757 72.9269 ) and LINESTRING ( -1.8263900000000035 71.5222, -180.0 71.72491214102453 ) [ (-133.39638594926308, 71.67189015654711, NaN) ]"}],"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"topology_exception","reason":"topology_exception: found non-noded intersection between LINESTRING ( -134.351 71.3427, -129.757 72.9269 ) and LINESTRING ( -1.8263900000000035 71.5222, -180.0 71.72491214102453 ) [ (-133.39638594926308, 71.67189015654711, NaN) ]"}},"status":400}
		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:326)
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:296)
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270)
		at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2082)
		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1732)
		... 48 more
Caused by: ElasticsearchException[Elasticsearch exception [type=topology_exception, reason=topology_exception: found non-noded intersection between LINESTRING ( -134.351 71.3427, -129.757 72.9269 ) and LINESTRING ( -1.8263900000000035 71.5222, -180.0 71.72491214102453 ) [ (-133.39638594926308, 71.67189015654711, NaN) ]]]
	at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:485)
	at org.elasticsearch.ElasticsearchException.fromXContent(ElasticsearchException.java:396)
	at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:426)
	at org.elasticsearch.ElasticsearchException.failureFromXContent(ElasticsearchException.java:592)
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:168)
	... 51 more

I update issue title

@suberti-ads suberti-ads changed the title [BUG] [Distribution v1.13.1-rc1+WA982] Fails to extract 4 OL1_EFR products generated by PUG [BUG] [Distribution v1.13.1-rc1+WA982] Fails to extract OL1 / OL2 products generated by PUG Jul 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pcuq-ads pcuq-ads

Labels
bug Something isn't working CCB Issue for CCB checked ops Ticket from ADS operation team priority:minor Set the priority to minor because the production is (almost) not impacted S3 Relative to Sentinel-3 RS Addons to_be_fixed_phase1 Issue to be fixed for RS phase 1
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@Woljtek @pcuq-ads @w-jka @suberti-ads @LAQU156