Skip to content

EcrfDataEntryTest testsuite#262

Merged
rkrenn merged 87 commits intomasterfrom
selenium_ecrfdataentry
Feb 23, 2023
Merged

EcrfDataEntryTest testsuite#262
rkrenn merged 87 commits intomasterfrom
selenium_ecrfdataentry

Conversation

@rkrenn
Copy link
Copy Markdown
Collaborator

@rkrenn rkrenn commented Feb 7, 2023

this is the commit for our 2nd End-To-End testsuite based on chrome webdriver/selenium server.
it is to be executed automatically by the GitHub java CI (see "Actions" https://github.com/phoenixctms/ctsms/actions).

the use case covered is the "eCRF Date Entry and Export", as raised by AGES Inspektion KPAM 398.

Test outline:

  • create phoenix instance, preload it with generated demo data
  • setup records required for the test:
    • department
    • user
    • trial
  • open the phoenix web UI in a chrome browser instance to programatically
    • log in
    • import sample eCRFs from a predefined test setup: ecrf.xls
      • cover all support field types
      • series sections
      • form scripting, calculated values
      • a new spreadsheet "validation" allows the specify what values to enter in the UI, and expect in the exported data
    • create a subject and add it to the trial's subject list
    • fill in each eCRF, with the data provided in the "validation" spreadsheet
    • run the eCRF data export job
    • download the exported data in SQLite format and verify the exported data:
      • compare each exported value with the reference data provided in the "validation" spreadsheet
      • check if all expected values are present
      • check if no unexpected values listed
  • generate a test report and send results via email
  • upload the test reports to an FTP target forarchiving
    • the test reports for the given commit can also be linked in the login screen for download

An example of the generated test report PDF is attached below. It represent the documentation mandatory for software validation in regulated environments.

Trial Tests.pdf

Feel free to contact us if you need these test report documents as well.

Summary by CodeRabbit

  • New Features

    • Status bar now links to validation reports when a URL is configured.
    • Added configurable setting for validation reports URL.
  • Improvements

    • Enhanced XLS import/export foundations for greater reliability.
  • Bug Fixes

    • Corrected “error occured” to “error occurred” in various status messages.
  • Tests

    • Added end-to-end eCRF data entry and validation test.
    • Improved test reporting with optional FTP upload of attachments.
  • Chores

    • Updated CI workflows, environment setup scripts, and dependencies (including Chromedriver and libraries).

@rkrenn rkrenn merged commit 14a7c8b into master Feb 23, 2023
@rkrenn rkrenn mentioned this pull request Nov 24, 2023
@rkrenn
Copy link
Copy Markdown
Collaborator Author

rkrenn commented Aug 20, 2025

@CodeRabbit review

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Aug 20, 2025

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Aug 20, 2025

Caution

Review failed

The pull request is closed.

Walkthrough

Updates CI scripts and Selenium setup; introduces Bulk Processor setup. Refactors XLS import/export with new base classes and JXL dependency, changing packages and interfaces. Adjusts CSV/XLS processors to use ProcessorJobOutput and public postProcess. Enhances tests, adds new trial ECRF Selenium test and FTP uploads. Adds settings and UI link for validation reports.

Changes

Cohort / File(s) Summary
CI scripts and workflow paths
.github/workflows/dbtool.sh, .github/workflows/init_database.sh, .github/workflows/test.yml
Fix CTSMS jar to 1.8.1; move dbtool.sh invocations to /ctsms/dbtool.sh; add BulkProcessor setup step; adjust permissions; load demo data via new path; extend test env vars and add extra Selenium test run.
Selenium setup and runner
.github/workflows/setup_selenium.sh, .github/workflows/run_selenium_test.sh
Switch Chromedriver download from 108 to 109; add Maven system properties for FTP and resource path to test run.
Tomcat service tweaks
.github/workflows/setup_tomcat.sh
Add tomcat to ctsms group; expand ReadWritePaths to include /ctsms/**, bulk output, and /tmp/.
Bulk Processor bootstrap
.github/workflows/setup_bulkprocessor.sh
New script to install dependencies, patch ImageMagick, install CPAN modules, fetch and unpack Bulk Processor to /ctsms/bulk_processor, set permissions, fetch helper scripts, and enable public publishing.
Top-level and module POMs
pom.xml, common/pom.xml, core/pom.xml
Manage commons-net:3.9.0; set application.uuid=test. Add jxl to common. In core: add sqlite-jdbc, commons-net, and duplicate selenium-chrome-driver test dep.
New ProcessorJobOutput API
common/src/main/java/.../fileprocessors/ProcessorJobOutput.java, core/src/exec/java/.../util/JobOutput.java
Add new interface with println(String); make JobOutput implement ProcessorJobOutput; minor message typo fix.
CSV processor package/type updates
common/src/main/java/.../fileprocessors/csv/LineProcessor.java, LineWriter.java
Move packages from executable.* to fileprocessors.*; replace JobOutput with ProcessorJobOutput; make postProcess() public.
XLS processor foundations
common/src/main/java/.../fileprocessors/xls/RowProcessor.java, RowWriter.java, InputfieldRowProcessorBase.java
Move packages to fileprocessors.xls; switch to ProcessorJobOutput; public postProcess(); add empty base for input-field row processors.
XLS import/export bases and contexts
common/src/main/java/.../fileprocessors/xls/XlsExporterBase.java, XlsImporterBase.java, XlsExporterContext.java, XlsImporterContext.java
New base classes orchestrating JXL-based print/read flows; contexts updated to use new base types; add jobOutput wiring and in-memory/file output handling.
Core XLS exporter/importer refactor
core/src/exec/java/.../xls/XlsExporter.java, XlsImporter.java
Refactor to extend new base classes; delegate IO/workbook handling; add setJobOutput(JobOutput) to exporter and email-attachment hook; importer overrides getInputStream(...) and removes local row-reading implementation.
Core CSV/XLS implementers visibility/import fixes
core/src/exec/java/.../csv/*LineProcessor.java, .../csv/*Writer.java, .../csv/CsvExporter.java, .../csv/CsvImporter.java, .../xls/*RowProcessor.java, .../xls/*RowWriter.java
Add imports for moved types; change postProcess() overrides from protected to public; add casts where contexts now expose base types. No logic changes beyond type access.
Core utilities and minor fixes
core/src/main/java/.../util/CoreUtil.java, .../util/date/DateCalc.java, core/src/main/resources/*.properties, core/src/main/java/.../trial/TrialServiceImpl.java
Aggregate full error stream in runProcess; fix “occurred” typos in messages; adjust parseFormatLongAndIncrement to use primitives and move increment post-parse.
Test infrastructure and beans
core/src/test/java/.../SeleniumTestBase.java, .../TestDataProvider.java, core/src/test/resources/applicationContextTest.xml
Add FTP upload of reports, resource helpers, UI utilities, and broadened method signatures (form-scoped). Add Proband DAO helper. Register beans for test XlsImporter and EcrfValidationRowProcessor.
New ECRF validation test support
core/src/test/java/.../selenium/trial/EcrfDataEntryTest.java, .../test/EcrfValidationTestVector.java, .../test/xls/EcrfValidationRowProcessor.java, .../test/xls/XlsImporter.java
Add end-to-end Selenium test implementing ProcessorJobOutput; add vector model; add XLS row processor for validation sheet; add test XlsImporter built on new base.
Web settings and UI link
web/src/main/java/.../ApplicationScopeBean.java, .../web/util/DefaultSettings.java, .../web/util/SettingCodes.java, web/src/main/resources/.../settings.properties, web/src/main/webapp/META-INF/templates/baseTemplate.xhtml
Introduce validation_reports_url setting and getter; default added; properties updated with URL; base template conditionally renders application instance as link when URL present.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  participant Caller as Caller
  participant Exporter as XlsExporter (extends Base)
  participant Base as XlsExporterBase
  participant Writer as RowWriter
  participant JXL as JXL Workbook

  Caller->>Exporter: setJobOutput(JobOutput)
  Caller->>Exporter: printRows(context, writer)
  Exporter->>Base: printRows(context, writer)
  Base->>Writer: setContext, init()
  alt file output
    Base->>JXL: create Workbook (File)
  else memory output
    Base->>JXL: create Workbook (ByteArray)
  end
  Base->>Writer: printHeaderRow()
  loop rows
    Base->>Writer: printRows()
  end
  Base->>JXL: write() and close()
  alt memory output
    Base->>Exporter: addEmailXlsAttachment(data)
    Exporter-->>Caller: attachment via JobOutput
  end
  Base-->>Caller: return row count
Loading
sequenceDiagram
  autonumber
  participant Test as SeleniumTestBase
  participant Sys as System Props
  participant FTP as FTP Server

  Test->>Test: generate report files (PDF/ZIP)
  Test->>Sys: read ftp.host, ftp.username, ftp.password, ftp.path
  Test->>FTP: connect/login
  loop each report file
    Test->>FTP: storeFile(remotePath, file)
  end
  Test->>FTP: logout/disconnect
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60–90 minutes

Possibly related PRs

Suggested reviewers

  • SigridDeller
  • ultchagen

Poem

Hop-hop, I stitch the sheets to rows,
XLS winds where CSV flows.
Scripts align, the drivers shine,
Reports now ride an FTP line.
A link appears where settings bloom—
Validation stars light up the room.
Thump-thump! Ship it soon. 🐇✨

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 ast-grep (0.38.6)
core/src/main/java/org/phoenixctms/ctsms/service/trial/TrialServiceImpl.java

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

💡 Knowledge Base configuration:

  • MCP integration is disabled by default for public repositories
  • Jira integration is disabled by default for public repositories
  • Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between e23dbb4 and f51a077.

⛔ Files ignored due to path filters (4)
  • core/src/exec/resources/ecrfs.xls is excluded by !**/*.xls
  • core/src/exec/resources/sdf-crf.xls is excluded by !**/*.xls
  • core/src/test/resources/ecrfs.xls is excluded by !**/*.xls
  • core/src/test/resources/input_fields.xls is excluded by !**/*.xls
📒 Files selected for processing (66)
  • .github/workflows/dbtool.sh (1 hunks)
  • .github/workflows/init_database.sh (1 hunks)
  • .github/workflows/run_selenium_test.sh (1 hunks)
  • .github/workflows/setup_bulkprocessor.sh (1 hunks)
  • .github/workflows/setup_selenium.sh (1 hunks)
  • .github/workflows/setup_tomcat.sh (1 hunks)
  • .github/workflows/test.yml (3 hunks)
  • common/pom.xml (1 hunks)
  • common/src/main/java/org/phoenixctms/ctsms/fileprocessors/ProcessorJobOutput.java (1 hunks)
  • common/src/main/java/org/phoenixctms/ctsms/fileprocessors/csv/LineProcessor.java (4 hunks)
  • common/src/main/java/org/phoenixctms/ctsms/fileprocessors/csv/LineWriter.java (2 hunks)
  • common/src/main/java/org/phoenixctms/ctsms/fileprocessors/xls/InputfieldRowProcessorBase.java (1 hunks)
  • common/src/main/java/org/phoenixctms/ctsms/fileprocessors/xls/RowProcessor.java (4 hunks)
  • common/src/main/java/org/phoenixctms/ctsms/fileprocessors/xls/RowWriter.java (2 hunks)
  • common/src/main/java/org/phoenixctms/ctsms/fileprocessors/xls/XlsExporterBase.java (1 hunks)
  • common/src/main/java/org/phoenixctms/ctsms/fileprocessors/xls/XlsExporterContext.java (3 hunks)
  • common/src/main/java/org/phoenixctms/ctsms/fileprocessors/xls/XlsImporterBase.java (1 hunks)
  • common/src/main/java/org/phoenixctms/ctsms/fileprocessors/xls/XlsImporterContext.java (2 hunks)
  • core/pom.xml (2 hunks)
  • core/src/exec/java/org/phoenixctms/ctsms/executable/csv/AlphaIdLineProcessor.java (2 hunks)
  • core/src/exec/java/org/phoenixctms/ctsms/executable/csv/BankIdentificationLineProcessor.java (2 hunks)
  • core/src/exec/java/org/phoenixctms/ctsms/executable/csv/CountryLineProcessor.java (2 hunks)
  • core/src/exec/java/org/phoenixctms/ctsms/executable/csv/CriterionPropertyLineProcessor.java (2 hunks)
  • core/src/exec/java/org/phoenixctms/ctsms/executable/csv/CsvExporter.java (1 hunks)
  • core/src/exec/java/org/phoenixctms/ctsms/executable/csv/CsvImporter.java (1 hunks)
  • core/src/exec/java/org/phoenixctms/ctsms/executable/csv/MimeTypeLineProcessor.java (2 hunks)
  • core/src/exec/java/org/phoenixctms/ctsms/executable/csv/OpsCodeLineProcessor.java (2 hunks)
  • core/src/exec/java/org/phoenixctms/ctsms/executable/csv/PermissionDefinitionLineProcessor.java (2 hunks)
  • core/src/exec/java/org/phoenixctms/ctsms/executable/csv/PermissionDefinitionWriter.java (1 hunks)
  • core/src/exec/java/org/phoenixctms/ctsms/executable/csv/PermissionTemplateWriter.java (1 hunks)
  • core/src/exec/java/org/phoenixctms/ctsms/executable/csv/StreetLineProcessor.java (2 hunks)
  • core/src/exec/java/org/phoenixctms/ctsms/executable/csv/TitleLineProcessor.java (2 hunks)
  • core/src/exec/java/org/phoenixctms/ctsms/executable/csv/ZipLineProcessor.java (2 hunks)
  • core/src/exec/java/org/phoenixctms/ctsms/executable/xls/AspRowProcessor.java (2 hunks)
  • core/src/exec/java/org/phoenixctms/ctsms/executable/xls/EcrfFieldRowProcessor.java (4 hunks)
  • core/src/exec/java/org/phoenixctms/ctsms/executable/xls/EcrfFieldRowWriter.java (3 hunks)
  • core/src/exec/java/org/phoenixctms/ctsms/executable/xls/EcrfRowProcessor.java (4 hunks)
  • core/src/exec/java/org/phoenixctms/ctsms/executable/xls/EcrfRowWriter.java (3 hunks)
  • core/src/exec/java/org/phoenixctms/ctsms/executable/xls/InputFieldRowProcessor.java (4 hunks)
  • core/src/exec/java/org/phoenixctms/ctsms/executable/xls/InputFieldRowWriter.java (4 hunks)
  • core/src/exec/java/org/phoenixctms/ctsms/executable/xls/RandomizationListCodeRowProcessor.java (2 hunks)
  • core/src/exec/java/org/phoenixctms/ctsms/executable/xls/SelectionSetValueRowProcessor.java (2 hunks)
  • core/src/exec/java/org/phoenixctms/ctsms/executable/xls/SelectionSetValueRowWriter.java (1 hunks)
  • core/src/exec/java/org/phoenixctms/ctsms/executable/xls/XlsExporter.java (2 hunks)
  • core/src/exec/java/org/phoenixctms/ctsms/executable/xls/XlsImporter.java (3 hunks)
  • core/src/exec/java/org/phoenixctms/ctsms/util/JobOutput.java (3 hunks)
  • core/src/main/java/org/phoenixctms/ctsms/service/trial/TrialServiceImpl.java (1 hunks)
  • core/src/main/java/org/phoenixctms/ctsms/util/CoreUtil.java (2 hunks)
  • core/src/main/java/org/phoenixctms/ctsms/util/date/DateCalc.java (1 hunks)
  • core/src/main/resources/ctsms-ecrffieldstatustypes.properties (1 hunks)
  • core/src/main/resources/ctsms-ecrfvalidationstatusnames.properties (1 hunks)
  • core/src/main/resources/ctsms-jobstatusnames.properties (1 hunks)
  • core/src/test/java/org/phoenixctms/ctsms/TestDataProvider.java (4 hunks)
  • core/src/test/java/org/phoenixctms/ctsms/selenium/SeleniumTestBase.java (16 hunks)
  • core/src/test/java/org/phoenixctms/ctsms/selenium/proband/CandidateSelectionTest.java (6 hunks)
  • core/src/test/java/org/phoenixctms/ctsms/selenium/trial/EcrfDataEntryTest.java (1 hunks)
  • core/src/test/java/org/phoenixctms/ctsms/test/EcrfValidationTestVector.java (1 hunks)
  • core/src/test/java/org/phoenixctms/ctsms/test/xls/EcrfValidationRowProcessor.java (1 hunks)
  • core/src/test/java/org/phoenixctms/ctsms/test/xls/XlsImporter.java (1 hunks)
  • core/src/test/resources/applicationContextTest.xml (2 hunks)
  • pom.xml (2 hunks)
  • web/src/main/java/org/phoenixctms/ctsms/web/model/ApplicationScopeBean.java (1 hunks)
  • web/src/main/java/org/phoenixctms/ctsms/web/util/DefaultSettings.java (1 hunks)
  • web/src/main/java/org/phoenixctms/ctsms/web/util/SettingCodes.java (1 hunks)
  • web/src/main/resources/org/phoenixctms/ctsms/web/settings.properties (1 hunks)
  • web/src/main/webapp/META-INF/templates/baseTemplate.xhtml (1 hunks)
✨ Finishing Touches
  • 📝 Generate Docstrings
🧪 Generate unit tests
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch selenium_ecrfdataentry

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Status, Documentation and Community

  • Visit our Status Page to check the current availability of CodeRabbit.
  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant