Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Checkin huge IFC file #962

Open
tchegito opened this Issue Apr 11, 2019 · 9 comments

Comments

Projects
None yet
2 participants
@tchegito
Copy link

tchegito commented Apr 11, 2019

When I submit a huge IFC file (more than 750Mb) operation seems to never finish.

I noticed a high memory consumption during the process (especially in IfcStepStreamingDeserializer#mappedObject which could be replaced by fastutil's Int2LongArrayMap, I'll do a pull request about that) but the problem might be in the next phase: during the geometry generation.

Actually, I can see all the expected phase "Uploading / Storing data / Generating geometry". And during this last one, the cursor is nearly at the end, but stuck.

On the server, with a top, I can see that:
image

Problem seems to be with IfcGeomServer. We can see that the process has been going for +12 hours.

What can I do about this problem ?

@tchegito

This comment has been minimized.

Copy link
Author

tchegito commented Apr 11, 2019

Note: it just ended (after 13 hours) but there isn't any revision in the project.

@tchegito

This comment has been minimized.

Copy link
Author

tchegito commented Apr 11, 2019

The configuration described above was on a Tomcat 9 freshly installed on a powerful machine (8 cores, 32Gb RAM). After some other tests, it works on a development environment, launched with LocalDevBimServerStarter (Xmx at 8Gb, exactly like Tomcat) in a hour and a half.

I'm very confused !

@tchegito

This comment has been minimized.

Copy link
Author

tchegito commented Apr 11, 2019

Okay I got it.

On my environment development, IfcOpenShellPlugin was in 0.5.49-SNAPSHOT, and I installed a new server with latest version: 0.5.55. There must be a regression between the two versions.
I suspect following commits:

They are going together, and the last one changes the IfcOpenShell version, taking a more recent SHA1 commit.

So I don't know exactly what's wrong, but there sure is a bug introduced between commit 4380e1d (working one) and 33cbcc2 (not working) on IfcOpenShell.

@rubendel

This comment has been minimized.

Copy link
Member

rubendel commented Apr 11, 2019

Thanks for the very extensive bug report. One notable change between these versions is (indeed as you noticed) the conversion from float vertices to double vertices. So more memory is required and it will be slower, but 11 hours is a bit much ;).

Other changes have been implemented in IOS as well, for example layer slicing, which for some models can have some serious impact on processing time, not entirely sure when that was introduced. This is going to be a setting in BIMserver, but that will be in 1.5.133 probably.

Maybe @aothms has an idea already, but it's probably best to try and see whether disabled layerset slicing will have any positive effect in this case. In don't suppose you can share this model so we can test it here?

@tchegito

This comment has been minimized.

Copy link
Author

tchegito commented Apr 12, 2019

I guess with "layerset slicing", you're talking about IfcOpenShellEngine#applyLayerSets ? But I didn't understood how we can set this configuration parameter. In the code, I see that both "applyLayerSets" and "calculateQuantities" are matching a value from key "CALCULATE_QUANTITIES_SETTING" in the plugin configuration map.

For now, I'll just ship a forked BimServer for my client, with 2 modifications (awaiting for a fix in IfcOpenShell later):

  • increase version plugin anteriority (because I need 0.5.50 and the oldest on screen ins 0.5.53)
  • replace HashMap by fastutils (this is not the root cause of that issue, but that will speed things up)

But I'm sorry, I can't share my client's IFC file. If you need some log files to figure out what's going on, tell me how activate it in order to provide you some help.

@tchegito

This comment has been minimized.

Copy link
Author

tchegito commented Apr 12, 2019

And I want to insist on something I was maybe unclear on: with the latest version of IfcOpenShell (33cbcc2), it's not just really long. It doesn't work at all ! Without rolling back to the previous one (4380e1d), BimServer is unable to import my huge IFC file.

@rubendel

This comment has been minimized.

Copy link
Member

rubendel commented Apr 12, 2019

Those settings are there indeed, but not working yet.

I'll have a look at Int2LongArrayMap. Did you measure the speedup? I actually can't imagine this being a bottleneck, since it's basically always IOS first, then (disk) IO, then CPU/memory.

@tchegito

This comment has been minimized.

Copy link
Author

tchegito commented Apr 17, 2019

Actually, I'm not sure it's the bottleneck if you provide enough memory, but that's a great optimization, and rather simple.
When I did my first attempts to this huge checkin, that was with the latest version of IfcOpenShell, the one which failed to generate the geometry. When I saw the process stucked, I did a memory analysis and see that 33 millions objects in a map were holding 1.1Gb
image

With fastutils Int2LongArrayMap, it should be reduced to the size of native types (int => 4bytes, long => 8bytes), so a 33 millions map would take "only" 33*12 =0.396Gb nearly three times less. I confess that I didn't made another memory analysis to confirm this, because meanwhile, I solved my checkin problem with rollbacking IfcOpenShell.

But I definitely will, and paste it here next week.

@tchegito

This comment has been minimized.

Copy link
Author

tchegito commented Apr 17, 2019

After further tests, I can say that Int2LongArrayMap is finally not appropriate at all ! Memory consumption might be lower, but performance is horrible !
I found a better tradeoff with Int2LongOpenHashMap from fastutils, and here is the result:
image

For the same number of records in the map (33 millions), we have 402 653 296 bytes occupied. That's basically the memory I calculated before (3 times less) and performance are close to standard HashMap.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.