Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Schema guessing from file is currently not implemented in HTTP Server source #1637

Closed
obermeier opened this issue May 30, 2023 · 0 comments
Closed

Schema guessing from file is currently not implemented in HTTP Server source #1637

obermeier opened this issue May 30, 2023 · 0 comments
Assignees
@obermeier
Labels
backend Everything that is related to the StreamPipes backend bug Something isn't working connect Related to the `connect` module (adapters) java Pull requests that update Java code
Milestone
0.93.0

Comments

@obermeier
Copy link
Member

Body

The provided examplefile is currently not processed by the schema guesser.

StreamPipes Committer

I acknowledge that I am a maintainer/committer of the Apache StreamPipes project.
@obermeier obermeier added bug Something isn't working java Pull requests that update Java code connect Related to the `connect` module (adapters) backend Everything that is related to the StreamPipes backend labels May 30, 2023
@obermeier obermeier self-assigned this May 30, 2023
obermeier added a commit to obermeier/streampipes that referenced this issue May 30, 2023
@obermeier
Add implementation for schema guessing from the example file in the HTTP
a5c599a 
Server source.

fixes apache#1637
fixes apache#1518 for streampipes-extensions
@bossenti bossenti added this to the 0.93.0 milestone May 30, 2023
@obermeier obermeier mentioned this issue May 30, 2023
Closed
obermeier added a commit that referenced this issue Jun 1, 2023
@obermeier
Add implementation for schema guessing from the example file in the HTTP
1cb22eb 
resolves #1637
fixes #1518 for streampipes-extensions
@tenthe tenthe closed this as completed in 752289c Jun 2, 2023
tenthe added a commit that referenced this issue Jun 5, 2023
@tenthe @dominikriemer
@obermeier @bossenti @dependabot
Sp 1592 (#1647)
50199e0 
* [hotfix] Remove set adapters from extensions

* [hotfix] Remove old resource files

* [#1289] Remove data set model

* [#1289] Remove data set model from ui

* [#1289] Remove `adapterType` and fix tests

* [#1289] Remove class `SpDataSet`

* [#1289] Remove legacy code in class `AdapterDescription`

* [#1289] Remove class `AdapterStreamDescription`

* [#1289] Remove class `ProtocolStreamDescription`

* [hotfix] Remove class `AdapterDescription`

* [#1289] Very first version for new adapter interface

* [#1289] Started to migrate the API

* [#1289] Deactivate old protocol API

* [#1289] Remove old models from the ui

* [#1289] Remove old models from the ui

* [#1289] Fix icons for adapters

* [#1289] Fix minor bug

* [#1289] Fix maven build

* [#1289] WIP refactor `AdapterConfigurationBuilder`

* [#1289] Fix AdapterConfigurationBuilder

* [#1289] Fix AdapterConfigurationBuilder

* [#1289] Guess schema works now for NewMachineDataSimulatorAdapter

* [#1289] Add start adapter

* [#1289] First version of new machine data simulator works

* [#1289] Working on the first new parser

* [#1289] First version of declare parser description

* [WIP] Add adapter configuration test

* [#1289] First generic adapter works now

* [#1289] Add parser exception to json parser

* [#1289] Add csv parser

* [#1289] Change implementation for json parser

* [#1289] Add json array parsers

* Add runtime context to adapters (#1290)

* Extract interface of  (#1290)

* Refactor OPC-UA adapter to new model (#1290)

* Refactor S7 adapter (#1290)

* Refactor Modbus adapter (#1290)

* Refactor Ros adapter (#1290)

* [#1289] Add geo json parser

* [#1289] Add xml parser

* [#1289] Add image parser

* Refactor Kafka adapter (#1290)

* Refactor MQTT adapter (#1290)

* Refactor Nats adapter (#1290)

* Refactor Pulsar adapter (#1290)

* Refactor RocketMQ adapter (#1290)

* Refactor TubeMQ adapter (#1290)

* Refactor HTTP server adapter (#1290)

* Refactor HTTP Stream adapter (#1290)

* [WIP] Refactor Parser API

* [WIP] Refactor Parser API

* [#1289] Fix file stream e2e test

* [#1289] Fix tests and csv parser for timestamps

* [#1289] Fix e2e connect tests

* [hotfix] Fix build

* [hotfix] Deactivate Mqtt integration test

* [#1289] Remove old format implementation from backend

* [#1289] Remove old format implementation from backend

* [#1289] Remove old format implementation from ui

* [#1289] Fix kafka e2e test

* Add migration script for specific adapters (#1289)

* Add initial version to migrate generic adapters (#1289)

* Modify format migrators (#1289)

* [#1289] Migrate influxdb to new interface

* [#1289] Fix ROS bridge adapter

* [#1289] Rename specific-adapter-configuration.component

* [#1289] Remove old data model

* [#1289] Fix mqtt integration test

* [#1289] Update Pulsar integration test

* [#1289] Remove DebugAdapterSink which is no longer needed for integration tests

* [#1289] Fix kafka adapter on restart

* [#1289] Fix migrations for all parser

* [hotfix] Fix image parser for adapters

* [hotfix] Change StreamPipes version to fix build problem

* [#1289] Move adapters

* [hotfix] Remove duplicate licence headers

* [hotfix] Fix conversion of pipeline element templates

* Move PullAdapter to extensions-management and api modules (#1289)

* Refactor structure of extensions-api, add supplier to adapter interface (#1581)

* Rename AdapterInterface to StreamPipesAdapter (#1590)

* [hotfix] Fix checkstyle issues

* [hotfix] Fix checkstyle issue

* [hotfix] Fix checkstyle issue

* [hotfix] Fix checkstyle

* Migrate ISS adapter and Flic adapter (#1289)

* Migrate connect adapters to new model (#1289)

* Properly apply configs to influx adapter (#1289)

* [hotfix] Fix checkstyle

* fix: minor adaptions

* fix: remove data set artifacts

* fix: minor fixes

* [hotfix] Regenerate typescript model

* [#1592] Add first version for an io link adapter

* Refactor data processor and sink API (#1632) (#1633)

* chore: mark streampipes-wrapper-python as deprecated (#1623)

* Bump types-requests in /streampipes-client-python (#1588)

Bumps [types-requests](https://github.com/python/typeshed) from 2.30.0.0 to 2.31.0.0.
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-requests
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump mkdocstrings[python] in /streampipes-client-python (#1631)

* chore: add missing support of NATS as messaging protocol (#1628)

* Bump spring-boot.version from 3.0.6 to 3.1.0 (#1595)

Bumps `spring-boot.version` from 3.0.6 to 3.1.0.

Updates `spring-boot-properties-migrator` from 3.0.6 to 3.1.0
- [Release notes](https://github.com/spring-projects/spring-boot/releases)
- [Commits](spring-projects/spring-boot@v3.0.6...v3.1.0)

Updates `spring-boot-starter-jersey` from 3.0.6 to 3.1.0
- [Release notes](https://github.com/spring-projects/spring-boot/releases)
- [Commits](spring-projects/spring-boot@v3.0.6...v3.1.0)

Updates `spring-boot-starter-jetty` from 3.0.6 to 3.1.0
- [Release notes](https://github.com/spring-projects/spring-boot/releases)
- [Commits](spring-projects/spring-boot@v3.0.6...v3.1.0)

Updates `spring-boot-starter-oauth2-client` from 3.0.6 to 3.1.0
- [Release notes](https://github.com/spring-projects/spring-boot/releases)
- [Commits](spring-projects/spring-boot@v3.0.6...v3.1.0)

Updates `spring-boot-starter-undertow` from 3.0.6 to 3.1.0
- [Release notes](https://github.com/spring-projects/spring-boot/releases)
- [Commits](spring-projects/spring-boot@v3.0.6...v3.1.0)

Updates `spring-boot-starter-web` from 3.0.6 to 3.1.0
- [Release notes](https://github.com/spring-projects/spring-boot/releases)
- [Commits](spring-projects/spring-boot@v3.0.6...v3.1.0)

Updates `spring-boot-maven-plugin` from 3.0.6 to 3.1.0
- [Release notes](https://github.com/spring-projects/spring-boot/releases)
- [Commits](spring-projects/spring-boot@v3.0.6...v3.1.0)

---
updated-dependencies:
- dependency-name: org.springframework.boot:spring-boot-properties-migrator
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.springframework.boot:spring-boot-starter-jersey
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.springframework.boot:spring-boot-starter-jetty
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.springframework.boot:spring-boot-starter-oauth2-client
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.springframework.boot:spring-boot-starter-undertow
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.springframework.boot:spring-boot-starter-web
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.springframework.boot:spring-boot-maven-plugin
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Refactor data processor and sink API (#1632)

* Cleanup pom file (#1632)

* Migrate processors to API (#1632)

* [hotfix] Add runtime provider, update pom

* Fix bug in standalone wrapper (#1632)

* Fix instance provision in compatibility layer (#1632)

* [hotfix] Disable Maven cache in workflow file

* [hotfix] Fix failing build by reverting Flink version

* [hotfix] Add proper time selection to another cypress test (#1641)

* Fix bug in aggregation flink module

* Properly return output source and schema info

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Tim <50115603+bossenti@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore: remove Plc4xPassiveAdapter

* Add implementation for schema guessing from the example file in the HTTP

resolves #1637
fixes #1518 for streampipes-extensions

* [#1592] Add io link master id to event

* [#1592] Provide documentation for io link adapter

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Dominik Riemer <dominik.riemer@gmail.com>
Co-authored-by: Stefan Obermeier <st.obermeier@seeburger.de>
Co-authored-by: bossenti <bossenti@posteo.de>
Co-authored-by: Tim <50115603+bossenti@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
jjbiggins pushed a commit to jjbiggins/streampipes that referenced this issue Jul 5, 2023
@tenthe @dominikriemer
@obermeier @bossenti @dependabot @jjbiggins
Sp 1592 (apache#1647)
181794c 
* [hotfix] Remove set adapters from extensions

* [hotfix] Remove old resource files

* [apache#1289] Remove data set model

* [apache#1289] Remove data set model from ui

* [apache#1289] Remove `adapterType` and fix tests

* [apache#1289] Remove class `SpDataSet`

* [apache#1289] Remove legacy code in class `AdapterDescription`

* [apache#1289] Remove class `AdapterStreamDescription`

* [apache#1289] Remove class `ProtocolStreamDescription`

* [hotfix] Remove class `AdapterDescription`

* [apache#1289] Very first version for new adapter interface

* [apache#1289] Started to migrate the API

* [apache#1289] Deactivate old protocol API

* [apache#1289] Remove old models from the ui

* [apache#1289] Remove old models from the ui

* [apache#1289] Fix icons for adapters

* [apache#1289] Fix minor bug

* [apache#1289] Fix maven build

* [apache#1289] WIP refactor `AdapterConfigurationBuilder`

* [apache#1289] Fix AdapterConfigurationBuilder

* [apache#1289] Fix AdapterConfigurationBuilder

* [apache#1289] Guess schema works now for NewMachineDataSimulatorAdapter

* [apache#1289] Add start adapter

* [apache#1289] First version of new machine data simulator works

* [apache#1289] Working on the first new parser

* [apache#1289] First version of declare parser description

* [WIP] Add adapter configuration test

* [apache#1289] First generic adapter works now

* [apache#1289] Add parser exception to json parser

* [apache#1289] Add csv parser

* [apache#1289] Change implementation for json parser

* [apache#1289] Add json array parsers

* Add runtime context to adapters (apache#1290)

* Extract interface of  (apache#1290)

* Refactor OPC-UA adapter to new model (apache#1290)

* Refactor S7 adapter (apache#1290)

* Refactor Modbus adapter (apache#1290)

* Refactor Ros adapter (apache#1290)

* [apache#1289] Add geo json parser

* [apache#1289] Add xml parser

* [apache#1289] Add image parser

* Refactor Kafka adapter (apache#1290)

* Refactor MQTT adapter (apache#1290)

* Refactor Nats adapter (apache#1290)

* Refactor Pulsar adapter (apache#1290)

* Refactor RocketMQ adapter (apache#1290)

* Refactor TubeMQ adapter (apache#1290)

* Refactor HTTP server adapter (apache#1290)

* Refactor HTTP Stream adapter (apache#1290)

* [WIP] Refactor Parser API

* [WIP] Refactor Parser API

* [apache#1289] Fix file stream e2e test

* [apache#1289] Fix tests and csv parser for timestamps

* [apache#1289] Fix e2e connect tests

* [hotfix] Fix build

* [hotfix] Deactivate Mqtt integration test

* [apache#1289] Remove old format implementation from backend

* [apache#1289] Remove old format implementation from backend

* [apache#1289] Remove old format implementation from ui

* [apache#1289] Fix kafka e2e test

* Add migration script for specific adapters (apache#1289)

* Add initial version to migrate generic adapters (apache#1289)

* Modify format migrators (apache#1289)

* [apache#1289] Migrate influxdb to new interface

* [apache#1289] Fix ROS bridge adapter

* [apache#1289] Rename specific-adapter-configuration.component

* [apache#1289] Remove old data model

* [apache#1289] Fix mqtt integration test

* [apache#1289] Update Pulsar integration test

* [apache#1289] Remove DebugAdapterSink which is no longer needed for integration tests

* [apache#1289] Fix kafka adapter on restart

* [apache#1289] Fix migrations for all parser

* [hotfix] Fix image parser for adapters

* [hotfix] Change StreamPipes version to fix build problem

* [apache#1289] Move adapters

* [hotfix] Remove duplicate licence headers

* [hotfix] Fix conversion of pipeline element templates

* Move PullAdapter to extensions-management and api modules (apache#1289)

* Refactor structure of extensions-api, add supplier to adapter interface (apache#1581)

* Rename AdapterInterface to StreamPipesAdapter (apache#1590)

* [hotfix] Fix checkstyle issues

* [hotfix] Fix checkstyle issue

* [hotfix] Fix checkstyle issue

* [hotfix] Fix checkstyle

* Migrate ISS adapter and Flic adapter (apache#1289)

* Migrate connect adapters to new model (apache#1289)

* Properly apply configs to influx adapter (apache#1289)

* [hotfix] Fix checkstyle

* fix: minor adaptions

* fix: remove data set artifacts

* fix: minor fixes

* [hotfix] Regenerate typescript model

* [apache#1592] Add first version for an io link adapter

* Refactor data processor and sink API (apache#1632) (apache#1633)

* chore: mark streampipes-wrapper-python as deprecated (apache#1623)

* Bump types-requests in /streampipes-client-python (apache#1588)

Bumps [types-requests](https://github.com/python/typeshed) from 2.30.0.0 to 2.31.0.0.
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: types-requests
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump mkdocstrings[python] in /streampipes-client-python (apache#1631)

* chore: add missing support of NATS as messaging protocol (apache#1628)

* Bump spring-boot.version from 3.0.6 to 3.1.0 (apache#1595)

Bumps `spring-boot.version` from 3.0.6 to 3.1.0.

Updates `spring-boot-properties-migrator` from 3.0.6 to 3.1.0
- [Release notes](https://github.com/spring-projects/spring-boot/releases)
- [Commits](spring-projects/spring-boot@v3.0.6...v3.1.0)

Updates `spring-boot-starter-jersey` from 3.0.6 to 3.1.0
- [Release notes](https://github.com/spring-projects/spring-boot/releases)
- [Commits](spring-projects/spring-boot@v3.0.6...v3.1.0)

Updates `spring-boot-starter-jetty` from 3.0.6 to 3.1.0
- [Release notes](https://github.com/spring-projects/spring-boot/releases)
- [Commits](spring-projects/spring-boot@v3.0.6...v3.1.0)

Updates `spring-boot-starter-oauth2-client` from 3.0.6 to 3.1.0
- [Release notes](https://github.com/spring-projects/spring-boot/releases)
- [Commits](spring-projects/spring-boot@v3.0.6...v3.1.0)

Updates `spring-boot-starter-undertow` from 3.0.6 to 3.1.0
- [Release notes](https://github.com/spring-projects/spring-boot/releases)
- [Commits](spring-projects/spring-boot@v3.0.6...v3.1.0)

Updates `spring-boot-starter-web` from 3.0.6 to 3.1.0
- [Release notes](https://github.com/spring-projects/spring-boot/releases)
- [Commits](spring-projects/spring-boot@v3.0.6...v3.1.0)

Updates `spring-boot-maven-plugin` from 3.0.6 to 3.1.0
- [Release notes](https://github.com/spring-projects/spring-boot/releases)
- [Commits](spring-projects/spring-boot@v3.0.6...v3.1.0)

---
updated-dependencies:
- dependency-name: org.springframework.boot:spring-boot-properties-migrator
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.springframework.boot:spring-boot-starter-jersey
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.springframework.boot:spring-boot-starter-jetty
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.springframework.boot:spring-boot-starter-oauth2-client
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.springframework.boot:spring-boot-starter-undertow
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.springframework.boot:spring-boot-starter-web
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.springframework.boot:spring-boot-maven-plugin
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Refactor data processor and sink API (apache#1632)

* Cleanup pom file (apache#1632)

* Migrate processors to API (apache#1632)

* [hotfix] Add runtime provider, update pom

* Fix bug in standalone wrapper (apache#1632)

* Fix instance provision in compatibility layer (apache#1632)

* [hotfix] Disable Maven cache in workflow file

* [hotfix] Fix failing build by reverting Flink version

* [hotfix] Add proper time selection to another cypress test (apache#1641)

* Fix bug in aggregation flink module

* Properly return output source and schema info

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Tim <50115603+bossenti@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore: remove Plc4xPassiveAdapter

* Add implementation for schema guessing from the example file in the HTTP

resolves apache#1637
fixes apache#1518 for streampipes-extensions

* [apache#1592] Add io link master id to event

* [apache#1592] Provide documentation for io link adapter

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Dominik Riemer <dominik.riemer@gmail.com>
Co-authored-by: Stefan Obermeier <st.obermeier@seeburger.de>
Co-authored-by: bossenti <bossenti@posteo.de>
Co-authored-by: Tim <50115603+bossenti@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@obermeier obermeier

Labels
backend Everything that is related to the StreamPipes backend bug Something isn't working connect Related to the `connect` module (adapters) java Pull requests that update Java code
Projects
None yet
Milestone
0.93.0
2 participants
@obermeier @bossenti