Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implementation and documentation of multi-threaded routing optimization #64

Merged
merged 10 commits into from
Sep 19, 2021
Merged

Implementation and documentation of multi-threaded routing optimization #64

merged 10 commits into from
Sep 19, 2021

Conversation

UAV-Pilot
Copy link

The implementation and usage are documented in README.md

@andrasfuchs
Copy link
Collaborator

I'll need a few days to test the new parameters and check their results, but it looks good, excellent contribution, thank you @UAV-Pilot!

@lynchaj
Copy link

lynchaj commented Sep 7, 2021 via email

@andrasfuchs
Copy link
Collaborator

@lynchaj I'm not familiar with the build process here on GitHub, so I can't help you with the 32-bit question from the top of my head. We moved to Java 11 so you can't compile it for Java 8, but building a 32-bit version of the Java 11 executable should be possible, I think, but again I'm not an expert.

Could you try to clone @UAV-Pilot's freerouting repo, compile the source code and run it on your computer locally? It should work on 32-bit Windows too.

@miho, what do you think?
Since it was asked before, could we add 32-bit executables to our next release after this PR gets merged?

andrasfuchs
andrasfuchs requested changes Sep 7, 2021
View reviewed changes
Copy link
Collaborator

@andrasfuchs andrasfuchs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1, I run the same test input .DNS file with various -mt settings:
a, without -mt (effectively using the default -mt 4),
b, -mt 128 and
c, -mt 1.
I got exactly the same .SES files with them and they all completed in 254-255 seconds. That is a little strange, isn't it? I got an 8-core CPU.
The original v1.4.4 run in 612 seconds with the same input, and produced a slightly different .SES file.
Do you know what the reason can be?

2, Freerouter produced some temp files and they are not deleted after the successful run. Could you add a cleanup procedure?
image

src/main/java/eu/FR.java Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
src/main/java/eu/mihosoft/freerouting/gui/StartupOptions.java Show resolved Hide resolved
src/main/java/eu/mihosoft/freerouting/gui/StartupOptions.java Show resolved Hide resolved
src/main/java/eu/mihosoft/freerouting/logger/FRLogger.java Outdated
@@ -10,7 +11,10 @@
import java.util.HashMap;

public class FRLogger {
private static Logger logger = LogManager.getLogger(FreeRouting.class);
private static Logger logger = LogManager.getLogger(FR.class);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I see what you did here. Alright, now I understand why you needed that empty class.
Although I fully agree with you that the original name of the class FreeRouting is too long and redundant, there must be a better way to hide it by configuring the logger.
This looks like a quick fix, so please revert it back.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Name "FreeRouting" alone is not too long, but "eu.mihosoft.freerouting.FreeRouting" was too long in log and made it harder to read log messages on screen. BTW, the original class FreeRouting is also an empty class. I actually have not worked with Java for quite long time except this project, so I'm not sure if it's easy to customize log4j for this behavior. If you're sure log4j can be customized easily for our purpose here, I'll revert it.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's reverted anyway. Someone please take care of it later.

@UAV-Pilot
Copy link
Author

With 8-cores, it's likely not meaningful to have more than 16 threads assuming each core is capable of hyperthreading. With such a simple board, the time for each thread to duplicate board is probably a large portion of execution time. Note that in README.md, I mentioned future enhancement about cloning.

It's better to use a more complex board to do the comparison test, e.g., a board previously needed more than 10 hours to optimize.

@lynchaj
Copy link

lynchaj commented Sep 11, 2021 via email

@lynchaj
Copy link

lynchaj commented Sep 11, 2021 via email

@UAV-Pilot
Copy link
Author

An open source example board and documentation of approximate 27 times speedup with this board in README.md were checked in the last commit.

@andrasfuchs
Copy link
Collaborator

andrasfuchs commented Sep 13, 2021 •
edited

@UAV-Pilot Thanks for the updates! Could you send me any of those files that had such big improvement in DSN format, so that I can also test them?

The strange thing is so far that no matter how much threads I set to use (with the -mt) both the results and the time need are the same with my inputs.

@lynchaj Could you somehow send me your Z80 DUART board as a DSN file? I would like to test it.
If GitHub prevents you from doing so, please try to zip it, or upload to a file sharing site, and include the link here.
Thank you!

If you still want to join the testing, I would suggest you to use IntelliJ IDEA development environment, it works well for me. I'll continue running some test this week, so let me know if you need any help with the setup.

Also, since @miho seems to be not reacting to the 32-bit executable request, could you take a look and try to compile the source into a 32-bit version? I would start with the gradlew.bat, but I might be wrong.

@UAV-Pilot
Copy link
Author

@andrasfuchs The board file freerouting/tests/smoothieboard-v1.1_preOpt.bin was checked in, so it's available in my repository. As documented in README.md, this file was created in Freerouting v1.4.4, and bin format is native format of Freerouting, so Freerouting v1.4.4 should be able to open this file.

@UAV-Pilot
Copy link
Author

BTW, memory usage in route optimization of tests/smoothieboard-v1.1_preOpt.bin reached about 10 GB, so performance of route optimization for boards of medium to high complexity with 32-bit version will be affected by limited memory.

@andrasfuchs
Copy link
Collaborator

Thanks for the file and the heads up about memory usage!

The bin file might be native, yes, but I would like to test it like it was in a normal scenario, passing it as a command line parameter, that's why I would prefer the board as DSN. Don't worry though, if you can't share it with me in that format, I will test the multi-threading functionality with other boards I have.

I'll try to make the time for it this week, and get back to you.

@UAV-Pilot
Copy link
Author

I tried to import Spectra Session file into KiCad and then to export DSN file, but an error prevented the import, so there is an bug somewhere in KiCad or freerouting with regarding importing Spectra Session file. While doing it, I noticed that previously I unintentionally left a ground zone on the top layer, so I went on to further change the imported KiCad project. Here are additional changes to the KiCad project: 1) removed a ground zone on the the top layer. 2) changes to resolve all DRC violations: changed footprints for several solder jumpers, shifted several R, C components, and moved J10, which seems to be in an odd place after the import. Before this pull request, I never used or worked on smoothieboard, so I don't know why some connectors were arranged in that way. In my latest commit, I put the whole KiCad project in zip format into the test folder. Inside it, there is an exported DSN file where majority of connections are not routed. Hopefully this zip file together with the DSN file inside can provide a starting point for anyone who likes to use it for testing. Actually any other publicly available board with enough complexity can also be used in test.

@andrasfuchs
Copy link
Collaborator

I just run the following command:
freerouting -mp 32 -de .\tests\smoothieboard-5driver.dsn -do .\tests\smoothieboard-5driver.ses -mt 32

As I investigated the process it run only on one core (despite the -mt 32 parameter) and my CPU usage was only 10-12%.

Looking at the source code I would have expected to see the "Start to run optimization pass" in the logs, but it looks like the optimization didn't start.
@UAV-Pilot Do you have any good guess why? Limiting the number of passes (with -mp 32) could cause this?

I also got an error while I tried to import the SES file:
image
I can confirm that this error was not introduced by your pull request, it's a separate issue.

@UAV-Pilot
Copy link
Author

As mentioned in my previous comment, "there is an exported DSN file where majority of connections are not routed". You can also verify it via menu Info > Incompletes.

It looks like importing from EAGLE to KiCad caused the SES importing issue. I did not encounter this problem with non-imported projects.

@UAV-Pilot
Copy link
Author

Revised the way to estimate speedup in the last commit. Previously time for Freerouting v1.4.4 to reach final via count and final trace length was used in the calculation.

@UAV-Pilot
Copy link
Author

DSN file within the zip was also updated

@andrasfuchs
Copy link
Collaborator

andrasfuchs commented Sep 18, 2021 •
edited

Excellent, thank you. I'm playing around with the source code right now.

Could you get the latest commits in master into your pull request, so I can continue testing?
I found a bug that caused my previous confusion, but it looks like it's not in your pull request. It's related to the -mp parameter and I just want to make sure that your pull request will work as intended after it gets merged.

Thank you for your work and patience!

@UAV-Pilot
Copy link
Author

My latest commit is in the pull request. I just added a routed Smoothieboard (in bin format) which is ready for route optimization. It's smoothieboard-5driver_preOpt.bin in the zip file. After removing " (component
(place @Hole0 1854580 -557911 front 0)
(place @Hole1 863985 -557911 front 0)
(place @Hole2 928755 -1486280 front 0)
(place @hole3 1853310 -1537080 front 0)
)" (Hopefully these are just mounting holes)
from the SES file, it can be imported although it might contain problems resulted from the above deletion. The resulted KiCad PCB board is: smoothieboard-5driver_preOpt.kicad_pcb in the zip file. As with any KiCad PCB files, routes exported to DSN are considered fixed by Freerouting by default, so it's a dead end to load DSN for route optimization unless you know a way in Freerouting to work around this restriction.

@UAV-Pilot
Copy link
Author

BTW, I never used the -mp option as all the boards except the smoothieboard that I used with Freerouting are boards designed by myself, and I wanted to get best possible PBC boards, so I did not put limit on passes.

andrasfuchs
andrasfuchs approved these changes Sep 19, 2021
View reviewed changes
src/main/java/eu/mihosoft/freerouting/gui/BoardFrame.java

/**
* Saves the interactive settings and the design file to disk.
* Returns false, if the save failed.
*/
boolean save()
public boolean save(String surffix)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's a typo, surffix -> suffix.

@andrasfuchs andrasfuchs merged commit 99f311e into freerouting:master Sep 19, 2021
@andrasfuchs
Copy link
Collaborator

andrasfuchs commented Sep 21, 2021 •
edited

@UAV-Pilot I run some tests and your multi-threaded implementation works fine. I made a few small changes and fixed only a few small bugs. So I just want to thank you again for your contribution!

It would be great to see you contribute more, so don't hesitate to contact me if you feel like implementing another good idea of yours as a pull request.

(I also fixed the snapshot generator, so you can test the binaries from the Release page)

@UAV-Pilot
Copy link
Author

@andrasfuchs I was kind of being forced to make this optimization when it seemed impossible to finish route optimization for one of my boards with previous version. Thanks for keeping Freerouting available and improving it!

@AlexColello
Copy link

Out of curiosity, is there a paper describing the algorithm the multi-threaded optimizations are based on?

@UAV-Pilot
Copy link
Author

Currently all of document about the multi-threaded optimization are in README.md. Multi-threading is implemented via Java thread pool. Greedy board updating strategy is based on classic greedy algorithm. Prioritized item selection strategy is implemented via priority queue if I remember correctly. PCB routing optimization is complicated, and effectiveness of various implemented algorithms probably varies from board to board. If you're interested, please help improve existing ones and also add new ones.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrasfuchs andrasfuchs andrasfuchs approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@UAV-Pilot @andrasfuchs @lynchaj @AlexColello