Not able to scan archive in follow mode

[rpachon]

Hello,

I use FixArchiveScanner to scan my archive and when I put follow mode to false all is working fine, I got all messages in the archive and the scanner return correctly when finished.

But when I use follow = true, I get no message.

[RichardWarburton]

Have you got a series of steps or some code that reproduces this issue?

[rpachon]

Hello,
After having investigated I have observed 2 different behaviors:

• Real case is, I have an artio engine connected and receiving fix message. I use a FixArchiveScanner in the same process (same driver) and when follow is false I receive everything and then FixArchiveScanner close. When I use follow = true, I receive nothing.
• I isolate FixArchiveScanner in a different process (different driver than artio process) to give you the code and with follow false or true I have the same messages sent, except that follow = false stop the process and follow = true do not send new messages (live messages)

public static void main(String[] args) throws IOException {
		System.setProperty(CONTROL_CHANNEL_PROP_NAME, "aeron:udp?endpoint=localhost:7010");
		System.setProperty(CONTROL_RESPONSE_CHANNEL_PROP_NAME, "aeron:udp?endpoint=localhost:7020");

		System.out.println(AERON_DIRECTORY_NAME);
		System.out.println(AERON_ARCHIVE_DIRECTORY_NAME);

		final MediaDriver.Context context = new MediaDriver.Context()
				.threadingMode(SHARED)
				.dirDeleteOnStart(true)
				.aeronDirectoryName(AERON_DIRECTORY_NAME);

		final Archive.Context archiveContext = new Archive.Context()
				.threadingMode(ArchiveThreadingMode.SHARED)
				.idleStrategySupplier(CommonConfiguration::backoffIdleStrategy)
				//				.deleteArchiveOnStart(true)
				.aeronDirectoryName(AERON_DIRECTORY_NAME)
				.recordingEventsChannel(RECORDING_EVENTS_CHANNEL)
				.archiveDirectoryName(AERON_ARCHIVE_DIRECTORY_NAME);

		try (ArchivingMediaDriver driver = ArchivingMediaDriver.launch(context, archiveContext)) {
			final IntHashSet queryStreamIds = new IntHashSet();
			queryStreamIds.add(CommonConfiguration.DEFAULT_OUTBOUND_LIBRARY_STREAM);
			queryStreamIds.add(CommonConfiguration.DEFAULT_INBOUND_LIBRARY_STREAM);

			try (FixArchiveScanner fixArchiveScanner = new FixArchiveScanner(
					new FixArchiveScanner.Configuration()
							.aeronDirectoryName(AERON_DIRECTORY_NAME)
							.idleStrategy(CommonConfiguration.backoffIdleStrategy()))) {
				System.out.println("Scan archive");

				fixArchiveScanner.scan(
						IPC_CHANNEL,
						queryStreamIds,
						TestScanner::onMessage,
						null,
						true,
						EngineConfiguration.DEFAULT_ARCHIVE_SCANNER_STREAM
				);
			}
		}
	}

	private static void onMessage(FixMessageDecoder fixMessageDecoder, DirectBuffer directBuffer, int i, int i1, ArtioLogHeader artioLogHeader) {
		System.out.println(fixMessageDecoder.body());
	}

[RichardWarburton]

Hi, thanks for expanding a bit but in general if people have support contracts with my company then I'll happily spend time to investigate / debug / fix issues. If you're just submitting an issue on the bug tracker as a member of the public then in order to investigate it I feel that I really need a proper reproducing test case. A reproducing test case should mean the whole code that is needed to reproduce the issue, not a snippet, and clearly fails.

The idea with follow mode is that you follow an existing system that is live. In your example code you're starting up a new media driver for the specific purpose of running the FixArchiveScanner, so it's not going to "follow" an Engine because there's no Engine.

[rpachon]

Hello Richard,
Thank you for your answer and I understand your point.

I will write a small sample to reproduce the issue and post here in case of.

[rpachon]

Hello, I share the sample code to reproduce in case of.

As the code is not easy to copy/paste, I put the sample to reproduce this issue in a git repo:
https://github.com/rpachon/artio-sample

[RichardWarburton]

I tried the sample code in your repo and got the following exception, is this expected?

	at io.aeron.ClientConductor.onError(ClientConductor.java:228)
	at io.aeron.DriverEventsAdapter.onMessage(DriverEventsAdapter.java:115)
	at org.agrona.concurrent.broadcast.CopyBroadcastReceiver.receive(CopyBroadcastReceiver.java:116)
	at io.aeron.DriverEventsAdapter.receive(DriverEventsAdapter.java:68)
	at io.aeron.ClientConductor.service(ClientConductor.java:1112)
	at io.aeron.ClientConductor.awaitResponse(ClientConductor.java:1170)
	at io.aeron.ClientConductor.addExclusivePublication(ClientConductor.java:458)
	at io.aeron.Aeron.addExclusivePublication(Aeron.java:286)
	at uk.co.real_logic.artio.engine.RecordingCoordinator.track(RecordingCoordinator.java:267)
	at uk.co.real_logic.artio.protocol.Streams.dataPublication(Streams.java:71)
	at uk.co.real_logic.artio.engine.framer.FramerContext.<init>(FramerContext.java:91)
	at uk.co.real_logic.artio.engine.FixEngine.initFramer(FixEngine.java:347)
	at uk.co.real_logic.artio.engine.FixEngine.<init>(FixEngine.java:312)
	at uk.co.real_logic.artio.engine.FixEngine.launch(FixEngine.java:79)
	at artio.ArtioSample.main(ArtioSample.java:74)```

[rpachon]

Hello, I test again and I have not this exception ie:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.agrona.nio.TransportPoller (file:/home/stan/.m2/repository/org/agrona/agrona/1.11.0/agrona-1.11.0.jar) to field sun.nio.ch.SelectorImpl.selectedKeys
WARNING: Please consider reporting this to the maintainers of org.agrona.nio.TransportPoller
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
InitiatorSession{connectionId=6831855856884755456, sessionId=1, state=ACTIVE, sequenceIndex=6, lastReceivedMsgSeqNum=0, lastSentMsgSeqNum=1} connected
Scan Archive

I firstly start the quickfix acceptor and then I start artio initiator and I have the previous logs. And no message coming from archive as I am in follow = true.