Skip to content

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Questions about the development environment #692

Closed
wimjongman opened this issue Nov 6, 2021 · 39 comments
Closed

Questions about the development environment #692

wimjongman opened this issue Nov 6, 2021 · 39 comments
Labels

Comments

@wimjongman
Copy link
Contributor

Please post all your questions about the development environment here.

@wimjongman
Copy link
Contributor Author

@wimjongman Which "application" corresponds to the BIRT designer (= what happens when I execute the eclipse binary in above package)? I just assumed that it's the default (DesignerApplication) and then I would assume that it behaves in the same way

@

@wimjongman Which "application" corresponds to the BIRT designer (= what happens when I execute the eclipse binary in above package)? I just assumed that it's the default (DesignerApplication) and then I would assume that it behaves in the same way

@patric-r
You have to take a look at the available applications and try them out. It should not be hard to guess.

@patric-r
Copy link

patric-r commented Nov 6, 2021

It's a pity that you seem to refuse to answer my simple question:
Which application type corresponds to eclipse.exe of the last stable BIRT designer?

What does it make sense forcing me starting 20 application types when you know the answer or when it is somewhere configured? I started 3, but I don't see any difference.
And I made a guess: It must be default, "ReportDesigner". But I can't use it, because it's not working - have you tried creating a simple BIRT project?

We're planning contributing to this project, but we need your or the communities support and a perspective that this project is really alive again and to provide at least basic help/documentation of how to start developing. But compared to other open source projects, it seems to be much harder here...

@ruspl-afed
Copy link
Contributor

Which application type corresponds to eclipse.exe of the last stable BIRT designer

Sorry, I do not understand this question :(

Are you trying to start BIRT Designer from sources?

What I can see is 2 product manifests with "designer" in its id

<product name="BIRT Report Designer All-in-one" uid="org.eclipse.birt.report.designer.all" application="org.eclipse.ui.ide.workbench" version="4.9.0.qualifier" useFeatures="true" includeLaunchers="true">

and

<product name="BIRT Report Designer RCP" uid="org.eclipse.birt.report.designer.rcp" application="org.eclipse.birt.report.designer.ui.rcp.DesignerApplication" version="4.9.0.qualifier" useFeatures="true" includeLaunchers="true">

So what we are talking about?

@patric-r
Copy link

patric-r commented Nov 6, 2021

I mean this one:
https://download.eclipse.org/birt/downloads/drops/I-R1-4.8.0-201805221921/birt-report-designer-all-in-one-4.8.0-20180522-win32.win32.x86_64.zip

Starting eclipse.exe shows me that what I call the "Report designer" (if you use other terms, it would be nice to learn them that we use the same terms).
In the "Report Design" perspective I have (in default configuration) in the bottom left the Navigator view, where I can create "Report design projects", see all files etc.

My question is: What do I have to do in order to start exactly this "application" from the BIRT development environment?

@ruspl-afed
Copy link
Contributor

To start "All in One" you need to select

org.eclipse.birt.report.designer.ui.rcp.BIRT as a product
and
org.eclipse.ui.ide.workbench as an application

if something is not working as expected please open the dedicated issue to focus on concrete scenario

@wimjongman
Copy link
Contributor Author

It's a pity that you seem to refuse to answer my simple question: Which application type corresponds to eclipse.exe of the last stable BIRT designer?

I am not going to spoon-feed you, my friend. We learn from frustration and bumping our noses. :) Good to see your frustration level rise ;)

What does it make sense forcing me starting 20 application types when you know the answer or when it is somewhere configured? I started 3, but I don't see any difference.

There are only a couple of them in the org.eclipse.birt namespace. Also, you have to clean your configuration before you start a new application otherwise you will not see any difference. Before you ask me how to do that, take a good look at the launch configuration form.

If an application does not run, you have to examine the messages. Most of the time there is a missing plugin or there are too many plugins.

I will organize a teams session for Monday. We can do a Q&A there.

Thank you for your testing efforts. Your tests were a driver for the recent developments. Try to stay positive.

@hvbtup
Copy link
Contributor

hvbtup commented Nov 8, 2021

The eclipse.exe from birt-report-designer-all-in-one is the right one.
If you build from source, there will be a zip in build\birt-packages\birt-report-all-in-one\target\products.
This is the right one (not the zip in build\birt-packages\birt-report-all-in-one\target).
You extract this somewhere and - given that you have a matching JRE (Java version and correct 32 or 64 bit) - you can start eclipse.exe in that directory.
Personally, I use a CMD script for this which ensures a correct environment, but this is not strictly necessary.
This eclipse.exe is the IDE.
Your first step should be to

  1. Close the misleading welcome screen (which is suited for Java development),
  2. Create a fresh workspace, and then open the Report Design perspective,
  3. Create a new BIRT project "birt" (a directory where your rptdesign files go)
  4. Create e.g. a folder for your libraries. I prefer a directory "res" inside "birt".
  5. From the menu, choose Window / Preferences / Report Design / Resource append "/res" (corresponding to 4) so that it reads "/res".

Now you are ready to design reports.

  1. Now your

@hvbtup
Copy link
Contributor

hvbtup commented Nov 8, 2021

BTW in my start script for the BIRT IDE, I set some environment variables and then call

start /D%~dp0birt %BIRT_EXE% -data %~dp0 -clean -vm "%JAVA_HOME%\bin\javaw.exe" -vmargs -Djava.util.logging.config.file=%~dp0logging.properties
(in a single line of course)

The logging.properties is only because I use java.util.logging from inside scripts in my reports.

@wimjongman
Copy link
Contributor Author

I filed an issue to make life a little bit better for consumers: #694

@patric-r
Copy link

patric-r commented Nov 8, 2021

To start "All in One" you need to select

org.eclipse.birt.report.designer.ui.rcp.BIRT as a product and org.eclipse.ui.ide.workbench as an application

if something is not working as expected please open the dedicated issue to focus on concrete scenario

Thanks!
I was only able to choose between product and application configuration.
What I did: I chose org.eclipse.birt.report.designer.ui.rcp.BIRT project first, then switched the radio button to 'application' and chose org.eclipse.ui.ide.workbench - this finally started the right "application".
I need to do some more tests, but it looks like the original all-in-one designer package.

The viewer is not working, I know this had been discussed recently - is the github issue #649 the right one for this (because I don't get any stacktrace, just a 404 in the browser when doing a PDF preview)?

And BTW, I saw this in the error log: Project facet group group.birt.runtime has not been defined. It is used in plugin org.eclipse.birt.chart.integration.wtp.ui.

@patric-r
Copy link

patric-r commented Nov 8, 2021

@wimjongman
Thanks for your tutorial how to run the junit tests in the other issue!

I have question regarding the last step, making the plugin configuration for the tests identical to the birt-run launch configuration:
Is there a shortcut to export this from birt-run and import it in the test configuration (ideally, we would have a pre-packaged launch configuration for the tests as well).
What about the checkboxes below the plugin list?

@hvbtup
Copy link
Contributor

hvbtup commented Nov 8, 2021

I finally (think that) I understand what you mean.
Your question was not about how to start the BIRT IDE as a stand-alone program. Instead you meant how to start the BIRT IDE from inside the Eclipse IDE in order to debug e.g. changes to emitter plugins or whatever. Right?

@patric-r
Copy link

patric-r commented Nov 8, 2021

I finally (think that) I understand what you mean. Your question was not about how to start the BIRT IDE as a stand-alone program. Instead you meant how to start the BIRT IDE from inside the Eclipse IDE in order to debug e.g. changes to emitter plugins or whatever. Right?

Yes :) Sorry for the confusion.
Because that's what I need when doing changes/fixes - to test them locally.
The all-in-one-designer is perfect for this, because I can test everything (from design to 'emitter')
within one application.

How do you test/debug your local working copy changes?

@wimjongman
Copy link
Contributor Author

Is there a shortcut to export this from birt-run and import it in the test configuration

I don't think there is a shortcut. The shortest way is maybe editing the file in a text editor. Launch configs are a moving target because of adding/removing future plugins. Launch configs can be different on different platforms. Therefore the developers should understand what makes them tick so that they can create/change them quickly.

@patric-r
Copy link

patric-r commented Nov 8, 2021

Especially because they are a moving target, it makes sense to commit them to the repo.
Otherwise, we need to rely on manual synchronization which will lead to increased effort (how do I know I need to adjust my plugins configuration?) and the developer needs to have a deeper understanding of the eclipse and BIRT ecosystem just to fix a trivial bug which is a hurdle for new contributors (like me for example).

@wimjongman
Copy link
Contributor Author

I see your POV, I really do, but being a BIRT maintainer without having basic Eclipse knowledge is not realistic.

Everything we put in the repo we also have to maintain. With a limited set of committers, it will be very hard to keep all of these things synchronized. e.g. see #693

I have started an Eclipse course recently that ppl might want to check out:

https://www.youtube.com/playlist?list=PLj9czDtxM-7w6Oc_0lVip1O2Z95C6w1Sk

@hvbtup
Copy link
Contributor

hvbtup commented Nov 8, 2021

How do you test/debug your local working copy changes?

The very hard way: I build the source with mvn, then extract the ZIP, start eclipse, set up a workspace, import the project, run a report. With older versions I also managed to debug from within Eclipse, but similar to you, I either forgot how to do this or it did no longer work in the same way as in 4.2.1. It's a bit like developing in the old mainframe days, because the MVN build takes 1/2 hour every time. Makes you think a bit more before changing something :-)

@hvbtup
Copy link
Contributor

hvbtup commented Nov 8, 2021

It should be noted that, unfortunately, basic eclipse knowledge (and here I mean: including detailed knowledege about OSGI and plugins, products, features) is no longer something that can be assumed as "every Java developer knows this anyway".

@wimjongman
Copy link
Contributor Author

How do you test/debug your local working copy changes?

The very hard way:

Lol, that is terrible.

It should be noted that, unfortunately, basic eclipse knowledge (and here I mean: including detailed knowledege about OSGI and plugins, products, features) is no longer something that can be assumed as "every Java developer knows this anyway".

Every framework requires time to learn. It is not hard once you get it ;)

@merks
Copy link
Contributor

merks commented Nov 9, 2021

This isn't so much a question but rather an observation. If you enable freeze reporting (enabled by default in the committers package used by the Oomph setup), you will see many traces involving this:

	at org.eclipse.core.internal.resources.MarkerManager.visitorFindMaxSeverity(MarkerManager.java:640)
	at org.eclipse.core.internal.resources.MarkerManager.findMaxProblemSeverity(MarkerManager.java:302)
	at org.eclipse.core.internal.resources.Resource.findMaxProblemSeverity(Resource.java:997)
	at org.eclipse.jdt.ui.ProblemsLabelDecorator.getErrorTicksFromMarkers(ProblemsLabelDecorator.java:298)
	at org.eclipse.jdt.ui.ProblemsLabelDecorator.computeAdornmentFlags(ProblemsLabelDecorator.java:243)
	at org.eclipse.jdt.internal.ui.viewsupport.TreeHierarchyLayoutProblemsDecorator.computeAdornmentFlags(TreeHierarchyLayoutProblemsDecorator.java:69)
	at org.eclipse.jdt.internal.ui.packageview.PackageExplorerProblemsDecorator.computeAdornmentFlags(PackageExplorerProblemsDecorator.java:43)
	at org.eclipse.jdt.ui.ProblemsLabelDecorator.decorateImage(ProblemsLabelDecorator.java:170)
	at org.eclipse.jdt.internal.ui.viewsupport.JavaUILabelProvider.decorateImage(JavaUILabelProvider.java:136)
	at org.eclipse.jdt.internal.ui.packageview.PackageExplorerLabelProvider.getImage(PackageExplorerLabelProvider.java:137)
	at org.eclipse.jface.viewers.DelegatingStyledCellLabelProvider.getImage(DelegatingStyledCellLabelProvider.java:198)
	at org.eclipse.jface.viewers.DecoratingStyledCellLabelProvider.getImage(DecoratingStyledCellLabelProvider.java:171)
	at org.eclipse.jface.viewers.DelegatingStyledCellLabelProvider.update(DelegatingStyledCellLabelProvider.java:124)
	at org.eclipse.jface.viewers.DecoratingStyledCellLabelProvider.update(DecoratingStyledCellLabelProvider.java:134)
	at org.eclipse.jface.viewers.ViewerColumn.refresh(ViewerColumn.java:144

The ~100,000 warnings are not helpful for performance. I think this is made worse by the fact that projects contain nested projects, e.g., /org.eclipse.birt-parent contains the entire tree and each of the branches is also a project duplicated in the explorer that also contains a tree of projects, also duplicated in the explorer.

Because of this structure, doing a search will find matches in multiple copies of the same file.

I think it would be good to use resource filters to hide the contents of subprojects, but that's a fair bit of work to set up and I want to be sure that there is agreement to do this before I spend time doing that.

Here's what it looks like now:

image

It could be made to look like this:

image

with a filter like this in the .project file

	<filteredResources>
		<filter>
			<id>1636443535383</id>
			<name></name>
			<type>10</type>
			<matcher>
				<id>org.eclipse.ui.ide.multiFilter</id>
				<arguments>1.0-name-matches-false-true-[^.].*</arguments>
			</matcher>
		</filter>
	</filteredResources>

Or it could be made to look like this:

image

With a filter like this:

	<filteredResources>
		<filter>
			<id>1636443721891</id>
			<name></name>
			<type>30</type>
			<matcher>
				<id>org.eclipse.ui.ide.multiFilter</id>
				<arguments>1.0-projectRelativePath-matches-false-true-[^.].*/.*</arguments>
			</matcher>
		</filter>
	</filteredResources>

What do you folks think?

@ruspl-afed
Copy link
Contributor

Ed @merks we discussed that it could be more winning to not import these "technical parant" projects to the workspace at all (#695). The those who need it already knows where to get it and they can import them manually.
This way we can omit much more calculations.

@merks
Copy link
Contributor

merks commented Nov 9, 2021

Well, once you don't import them, then the pom.xmls become way less accessible. In combination with filtering and working sets one could provide access to all the things, like the pom.xmls, without duplicating the project/ folder/file structure. All the less interesting "technical parent" projects could of course be in a single working set where you can easily ignore it but also still easily access everything including and searching/editing all the poms...

E.g, I was playing with that like this, where I still need to determine what to do with the "Other Projects" projects:

image

I just don't want to do a bunch of work that in the end you guys don't actually want!

@wimjongman
Copy link
Contributor Author

We want that! Having the "group" folders for their content but hiding the actual embedded projects would be awesome.

The embedded projects can go in a working set with the same name as the parent folder?

https://user-images.githubusercontent.com/208716/140882114-38fb0a25-8521-4f8c-b90d-1da602073ced.png

@ruspl-afed
Copy link
Contributor

So, you idea is to move "technical parent" to a dedicated Working Set and then hide this Working Set?
I'm not sure that it will save us from additional calcultations during delta processing and other operations that has workspace scope - but this shoud definitely help from wasting resources in UI thread for decoration and similar activities.
Please proceed Ed if you have time.

@wimjongman
Copy link
Contributor Author

I think we have to remove all 'parent' projects from the workspace. Hiding them in the project explorer will not fool the rest of Eclipse.

I just manually removed all the parents but after a restart, the setup task was adding them again.

Also, it looks like the setup task is responsible for reloading the target platform which then leads to a complete recompile, which is unwanted after the first time.

@merks
Copy link
Contributor

merks commented Nov 10, 2021

I fixed the reloading of the target platform already:

https://git.eclipse.org/c/oomph/org.eclipse.oomph.git/commit/?id=9ac25ae3ecb1145ee7c148436e19ba3e36849bfd

That's only in a nightly build so far...

But even with that, there is still rebuild upon startup and it's not clear what causes that. It happens even when to setup tasks are performed...

As I have suggested the problem isn't parent projects themselves but rather the fact that they show/include the entire tree of nested projects in them which can be avoided with filters, which I have started to work on...

@hvbtup
Copy link
Contributor

hvbtup commented Nov 10, 2021

I tried the eclipse installer and opened the workspace as Wim showed in the Q&A session. Eclipse shows 487 errors and ~100K warnings. I dont' understand why on my machine there are 487 errors, because I saw only 14 (?) on Wim's machine.
Maybe would make sense to share my screen in the next Q&A session. I certainly missed some tiny but important step.

BTW, Wim wanted to share the recorded Q&A session.

@wimjongman
Copy link
Contributor Author

As I have suggested the problem isn't parent projects themselves but rather the fact that they show/include the entire tree of nested projects in them which can be avoided with filters, which I have started to work on...

Got it. I was confused about resource filters but I looked it up and now I'm good. I filed an issue for the resource filters:

Add resource filters to parent projects #699

@patric-r
Copy link

What is the recommend way / workflow to test own applications which use the BIRT runtimes againt the checked out working copy of BIRT without creating all BIRT jars by a full maven build and assigning those to the applications?

E.g. by adding "Required Projects" (birt projects) to the build path for such applications.
But which BIRT projects? How do you do it?

@ruspl-afed
Copy link
Contributor

ruspl-afed commented Nov 11, 2021

Assumptions:

  1. you have a .target for your BIRT-based product (that includes BIRT features/bundels)
  2. you have a workspace with you BIRT-based sources that uses this target
  3. you have launch configuration that runs your BIRD-based product

Steps:

  1. clone BIRT sources
  2. Import BIRT projects that you would like to see in source form (perhaps, to fix something?)
  3. normally, launch configuration will be auto switched from "target" to "workspace" instance of bundle but you can ensure that on "Plug-ins" tab of launch configuration.

@patric-r
Copy link

patric-r commented Nov 11, 2021

@ruspl-afed
Not quite. Our application is not an eclipse product. It's a plain java application. It doesn't have a GUI.
Currently, we add the following BIRT/eclipse jars to the classpath:

image

Afterwards, we start the platform, load the design handle and create the runAndRender task:

    Platform.startup(engineConfig_);
    this.reportEngine_ = new ReportEngine(engineConfig_);
    reportEngine_.changeLogLevel(Level.WARNING);
    this.pdfRenderOption_.setOutputFormat(FILE_EXTENSION);
    this.reportRunnable_ = reportEngine_.openReportDesign(this.reportDesign_);
    this.runAndRenderTask_ = reportEngine_.createRunAndRenderTask(reportRunnable_);		 

For each to-be-created PDF we do:

    this.runAndRenderTask_.setRenderOption(pdfRenderOption_);
    runAndRenderTask_.setLocale(Locale.GERMAN);
    this.runAndRenderTask_.run();

@hvbtup
Copy link
Contributor

hvbtup commented Nov 11, 2021

In our application we work similar to @patric-r. We are using the "birt-runtime" here (without OSGI). But instead of explicitly adding single JARs we just add ReportEngine/lib/* to the classpath. This includes all the files in that directory.

This requires the Maven build of course.

But to just test modifications of the BIRT source, we do not need to do it this way:
Our reports are written in such a way that they can be tested from within the BIRT designer (the DB connection info is supplied using environment variables then). So I can test from within Eclipse instead of from our application.
Soemtimes, when trying to fix a bug, I try to create a simple test case using only the builtin sample database. This makes testing the effect of source code changes even easier and allows other BIRT developers to see the effect as well.

@patric-r
Copy link

patric-r commented Nov 11, 2021

@hvbtup
Glad to hear that we're not the only one integrating BIRT that way.
Thanks for the info for the non-OSGI approach / BIRT runtime, I will definitely look into that.

You're completely right, we do it in a similar way that we can test everything in the BIRT designer (except that we don't use DB connections within BIRT, we decoupled the db model from the BIRT design model, so our process is roughly -> Export relevant data from DB to XML, start our application which invokes BIRT in the above way and the design uses the xml as a datasource).

However, I know that there are performance issues and memory leaks in the BIRT engine (at least in the version we're using) which only appear when creating many PDFs within one ReportEngine.
In order to fix them it is much more easier invoking the rendering task multiple times automatically instead of manually clicking 20 times on the preview button (and the viewer is currently not working).

Is there a way to achieve this?
I would love to improve memory consumption and maybe even rendering performance.

@merks
Copy link
Contributor

merks commented Nov 11, 2021

Note that JDT's increment compiler automatically creates the <project>/target/classes/ folder for each project so everywhere that you need a jar you could instead point at this corresponding folder...

@merks
Copy link
Contributor

merks commented Nov 11, 2021

Here's another trick to build a class path. Go to any of the JUnit tests and do Run as (or DebugAs) -> JUnit test. Then open the Debug perspective (upper right button with +), and you see the terminated process. Right click (context menu) the nested process object and use Properties... Here you can see the command line used to launch the test. This has the full class path of everything used by that test.

image

@wimjongman
Copy link
Contributor Author

We have created a video that explains the full cycle of contributing to BIRT.

https://youtu.be/FqfrG2I0AIw

00:00 Intro
00:40 Fork the main BIRT repository
02:00 Download the Eclipse Installer
03:10 Start Installation
05:20 Start Eclipse
05:50 Automatically configure the development environment
09:00 Running BIRT with your changes
12:00 Running Unit Tests
14:00 Making and pushing a change
16:00 Creating a Pull Request

@hvbtup
Copy link
Contributor

hvbtup commented Nov 11, 2021

However, I know that there are performance issues and memory leaks in the BIRT engine (at least in the version we're using) which only appear when creating many PDFs within one ReportEngine. In order to fix them it is much more easier invoking the rendering task multiple times automatically instead of manually clicking 20 times on the preview button (and the viewer is currently not working).

Is there a way to achieve this? I would love to improve memory consumption and maybe even rendering performance.

I don't know about BIRT specific memory leaks and performance issues.
But in the process model that we are running BIRT reports (sevearl Java processes, each one processes several reports single-threaded) I noticed that

  • BIRT needs quite a lot of memory when generating a complex PDF report. This is ok though.
  • So our JVM processes are created with eg -xmx 512m or 1024m (for larger reports) (don't know the exact syntax of the memory parameters right now)
  • The JVM seemts to perform a "big" GC only when the heap reaches its maximum (by default).
  • The GC then takes a lot of time
  • Having too low -xmx values causes the JVM to perform GC too often (e.g. moving a lot of small objects only to get tiny new bits of free space). This causes hight CPU load and slow reports.
  • Having too high -xmx values can cause out-of-memory errors on the OS side (or in Java) with this multi-process model because by default the JVM does not give back unused memory to the OS.
  • Before we switched to the multi-process report execution model, we used a mult-threaded single process. The reason why we switched was because some reports (e.g. containing many charts or images) need a lot of memory, while others only need a little bit. The little report may run into OutOfMemoryError when allocating a few dozen bytes because the big report allocated hundres of MB. Users don't understand this. And once you have OOM, the only thing you can safely do with your JVM is to exit the process. So one report could cause failures in other reports. This risk is minimized with the multi-process execution model.

In order to do load testing or performance testing, you should definitely use test scripts or programs with loops which start e.g. 1000 reports and wait until all 1000 are finished.

@colinsutton
Copy link

We run all reports with -d64 -Xmx6200m -Xms512m (though only one huge report needs the 6GB), using birt (4.5 and 4.8) under wildfly on dedicated windows servers with multiple users.

@hvbtup
Copy link
Contributor

hvbtup commented Nov 15, 2021

tried the eclipse installer and opened the workspace as Wim showed in the Q&A session. Eclipse shows 487 errors and ~100K warnings.

I was able to solve this. Now I only have one remaining error message:

Host bundle 'org.eclipse.birt.data.oda.pojo.ui' cannot be resolved

in org.eclipse.birt.data.oda.pojo.ui/META-INF/MANIFEST.mf

This seems to be a basic beginner's eclipse issue. I think it is worth mentioning.

What I had to do:

  1. Choose the JRE that came with the eclipse installer in Preferences / Java / Installed JREs

grafik

I have to repeat this every time I start eclipse.exe, as it seems. Otherwise Eclipse finds and uses the JRE 11 in c:\prog\jdk... and that causes a lot of "Failed to init ct.sym" errors.

For birt-runtime-test (inside BIRT Tests) and org.eclipse.birt.axis.overlay (inside BIRT Bundles) I had to explicitly choose JavaSE-11 in the build path.

@eclipse-birt eclipse-birt locked and limited conversation to collaborators Nov 15, 2021

This issue was moved to a discussion.

You can continue the conversation there. Go to discussion →

Labels
Projects
None yet
Development

No branches or pull requests

6 participants