Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ARM] Port OPNsense to the Netgate SG3100 #162

Closed
rene-bayer opened this issue Nov 6, 2019 · 31 comments
Closed

[ARM] Port OPNsense to the Netgate SG3100 #162

rene-bayer opened this issue Nov 6, 2019 · 31 comments
Labels
support Community support

Comments

@rene-bayer
Copy link
Contributor

Hello all,

i want to get opnsense on my Netgate SG3100 appliance.
(i learned a lot during this project ^^)

After hours of debugging, trying and compiling, i got an image, which boots with all the marvell driver loaded <3

But now i'm facing the problem, that OPNsense just gets unresponsive after successful boot :-(
image

Any ideas?

All the best,
René
@AdSchellevis AdSchellevis added the support Community support label Nov 6, 2019
@rene-bayer
Copy link
Contributor Author

when i add a simple "/bin/csh" here it drops me into a shell, and everything seems to work so far.

@fichtner
Copy link
Member

fichtner commented Nov 6, 2019

very strange spot to "hang".. did you configure auto-login on the console?

@rene-bayer
Copy link
Contributor Author

Next strange thing ... this only occurs over serial output.

Over SSH everything works as expected, seems to be a very device specific problem

@fichtner
Copy link
Member

fichtner commented Nov 6, 2019

It's likely /etc/ttys or /etc/login.conf related.

@nekoprog
Copy link
Member

nekoprog commented Nov 7, 2019

@DarkSunOne Can your serial detect input? I tried with usb-serial, but cannot send input to console.

@rene-bayer
Copy link
Contributor Author

rene-bayer commented Nov 7, 2019 •
edited
Loading

@nekoprog Yes, as said .. when i add a "/bin/csh" here, everything works, and i can configure everything.

@fichtner good hint with ttys ... i get the followig errors, when i reboot the box:
image

@rene-bayer
Copy link
Contributor Author

rene-bayer commented Nov 7, 2019 •
edited
Loading

So, is there anyone who is able to help me create a flashable image?

Currently i dd the arm image to a usb stick, and start it directly from this stickl (u-boot>> usb start ; boot)

I recieved a recovery image from netgate, but i have no idea, who to build a opnsense image like this :-(

It an .img file, this image file contains a 0.fat file, and a 1.img file.
0.fat seems to be the 1 stage bootloader, and 1.img seems to be the root system.

I can also provide this image to one of you if somone is interested or can help :-)

Edit:
this is what the recovery image looks like:
image

@fichtner
Copy link
Member

fichtner commented Nov 7, 2019

I can't speak for the image, but at least there's one of your issues: there's no ttyvX devices, maybe it's all USB-based, list your /dev/tty* to see what's there. :)

@nekoprog
Copy link
Member

nekoprog commented Nov 7, 2019 •
edited
Loading

So, is there anyone who is able to help me create a flashable image?

Currently i dd the arm image to a usb stick, and start it directly from this stickl (u-boot>> usb start ; boot)

I recieved a recovery image from netgate, but i have no idea, who to build a opnsense image like this :-(

It an .img file, this image file contains a 0.fat file, and a 1.img file.
0.fat seems to be the 1 stage bootloader, and 1.img seems to be the root system.

I can also provide this image to one of you if somone is interested or can help :-)

Edit:
this is what the recovery image looks like:
image

If you can create a tree for each of the item listed on 0.fat and 1.img, I can point you somewhere. OPNsense img also structured like that if extracted from .img. Where 0.fat is loaded to mmc/fat/MSDOSBOOT and 1.img is loaded to mmc/ufs/${ARMLABEL} on your flashed microsd card.

0.fat is where arm.sh copies /boot or ${STAGEDIR}/boot from OPNsense to fatmmc/boot/msdos or ${STAGEDIR}/boot/msdos, mind you that these had to mounted as of arm.sh, this is where the process starts for creating 0.fat.

1.img is where ${STAGEDIR} being converted into ufsmmc/ or basically the OS itself.

If you reverse the step, you can get the ${STAGEDIR} structure for Netgate recovery image.

@rene-bayer
Copy link
Contributor Author

Oh, okay ... well, didn't looked so close to the opnsense image yet ^^

Thanks for the explanation, so basically i just need to mount 1.img from the recovery image,
find the recovery image which starts on boot,
migrate that to the opnsense Souce and build a image with that 🤔

@rene-bayer
Copy link
Contributor Author

rene-bayer commented Nov 8, 2019 •
edited
Loading

Okay, i just made a successful flash, i replaced the built in pfsense image with my opnsense image.
(Far away from a ready to use in production)

It really boots up fine, but i recieve a lot of UFS and vfs errors:
UFS /dev/diskid/DISK-0A4A45A0s2a (/) cylinder checksum failed: cg 2, cgp: 0x5d474673 != bp: 0x2c1c94f6
and

mmcsd0: Error indicated: 2 Bad CRC
g_vfs_done():diskid/DISK-0A4A45A0s2a[WRITE(offset=1047527424, length=32768)]error = 5

And the opnsense shell is still not loaded (so no input is possible)

Feels so good, to have the goal so close <3
image

@rene-bayer
Copy link
Contributor Author

rene-bayer commented Nov 8, 2019 •
edited
Loading

@fichtner

I can't speak for the image, but at least there's one of your issues: there's no ttyvX devices, maybe it's all USB-based, list your /dev/tty* to see what's there. :)

Here we go (Image is now directly booted from mmc not usb):
image

@rene-bayer
Copy link
Contributor Author

rene-bayer commented Nov 8, 2019 •
edited
Loading

Next hint :)
as told before, i added /bin/csh at line 2 for debugging, so i can use the serial shell.

When i start opnsense-shell again (on the cli), everything is working, the problem only occurs with the "first" shell which starts directly after boot
image

@fichtner
Copy link
Member

fichtner commented Nov 8, 2019

ttyu0 and ttyu1 are there... can you check that /etc/ttys on the original image match ours? There's no "login:" prompt so it never finds the primary serial (likely ttyu0) maybe also due to some quirk device hint that needs to be set on the hardware. I really don't know. For details see i.e. https://forum.opnsense.org/index.php?topic=6998.msg31097#msg31097

@fichtner
Copy link
Member

fichtner commented Nov 8, 2019

Second thought.. have you tried to set the thing to serial on the GUI?

@fichtner
Copy link
Member

fichtner commented Nov 8, 2019

Reference: https://github.com/opnsense/tools/blob/master/config/20.1/extras.conf#L29-L37

arm was HDMI so not serial yet, nano has dual console, serial has serial console likewise

@nekoprog
Copy link
Member

nekoprog commented Nov 8, 2019 •
edited
Loading

Next hint :)
as told before, i added /bin/csh at line 2 for debugging, so i can use the serial shell.

When i start opnsense-shell again (on the cli), everything is working, the problem only occurs with the "first" shell which starts directly after boot
image

You might want to start debug from here and look what really happens after system_console_unmute().

Or maybe here since you already shown OPNsense banner. I hope you can find something there.

@nekoprog
Copy link
Member

nekoprog commented Nov 8, 2019

Reference: https://github.com/opnsense/tools/blob/master/config/20.1/extras.conf#L29-L37

arm was HDMI so not serial yet, nano has dual console, serial has serial console likewise

Can we merge arm and nano hook?

@rene-bayer
Copy link
Contributor Author

rene-bayer commented Nov 8, 2019 •
edited
Loading

Second thought.. have you tried to set the thing to serial on the GUI?
Yes, already tried all the settings :-(

will read the links later

...
Okay ... i fixed it but i dont like it :-D

When i remove "banner" here everything works as expected, at least it seams so.
Why is the banner called here, and not the "normal whole" opnsense-shell?

Edit: okay, well i see, the "program" is running in a while, so nothing is executed after that

@fichtner
Copy link
Member

fichtner commented Nov 8, 2019 via email

@fichtner
Copy link
Member

fichtner commented Nov 8, 2019 via email

@rene-bayer
Copy link
Contributor Author

rene-bayer commented Nov 8, 2019 •
edited
Loading

I still try to understand all the rc stuff from freebsd ...

when /usr/local/etc/rc is executed on boot, /usr/local/sbin/opnsense-shell banner is executed, this script just ends with exit code 0.
So everything is fine at this point, but theres nothing whats getting triggerd after that, i think we have to search in this direction...

Why is nothing executed after successfully finishing /usr/local/etc/rc?

Shouldn't there be a login prompt after that?

@fichtner
Copy link
Member

fichtner commented Nov 8, 2019 via email

@rene-bayer
Copy link
Contributor Author

Slowly this all make sense to me :-)

Tried with the following settings:
image
... still no luck :-(

@nekoprog
Copy link
Member

nekoprog commented Nov 8, 2019 •
edited
Loading

I still try to understand all the rc stuff from freebsd ...

when /usr/local/etc/rc is executed on boot, /usr/local/sbin/opnsense-shell banner is executed, this script just ends with exit code 0.
So everything is fine at this point, but theres nothing whats getting triggerd after that, i think we have to search in this direction...

Why is nothing executed after successfully finishing /usr/local/etc/rc?

Shouldn't there be a login prompt after that?

It's not about freebsd, but it's opnsense feature. It should not prompt you shell after successfull login, but you'll be prompted a cli dashboard to manage opnsense via cli console. There will be a menu to enter a shell, which you have to enter number 8 if I'm not mistaken.

It's same with pfsense.

For this purpose, I think you should try to start debug from here to find out why cli menu is not present after banner. Make sure to comment out this line. You could try change this to /usr/local/sbin/opnsense-shell and boot again, see if anything changes.

@rene-bayer
Copy link
Contributor Author

@nekoprog yes, if i change this line to /usr/local/sbin/opnsense-shell everything works as expected.
The interesting thing we need to troubleshoot is, why the login prompt isnt showing up.

As @fichtner already said, this should be /dev/tty* related, but i have no idea how to identify the problem at the moment.

@fichtner , other question, i have to implement the whole switch configuration in the gui, do you have tips for me where i should start?
These are completely new interfaces which i (or we) have to bring up in the gui.
On the FreeBSD site everything is set up and working, so its really only the OPNsense (+gui) -stuff
Should i create a separate issue for that?

@fichtner
Copy link
Member

fichtner commented Nov 8, 2019 via email

@rene-bayer
Copy link
Contributor Author

First i want to fix the open issues i am facing at the moment.

First, and most annoying is still that there's no login prompt.

I now know, that the pfsense image is definitely using /dev/ttyu0
and /dev/ttyu0 is also there in opnsense, so where to start troubleshooting?

@fichtner
Copy link
Member

@DarkSunOne it might be a good opportunity to ask the manufacturer what is needed to make that happen

@rene-bayer
Copy link
Contributor Author

So i should ask netgate, why opnsense doesn't show me a login prompt on there branded hardware? :-D

@fichtner
Copy link
Member

Why not, assuming FreeBSD has the same issue they know how to fix it.

@rene-bayer rene-bayer mentioned this issue Nov 13, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
support Community support
Milestone
No milestone
Development

No branches or pull requests
4 participants
@fichtner @nekoprog @AdSchellevis @rene-bayer